on 09-07-2012 2:26 PM
Hi All,
We are facing ine issue in daily P2P load from SRM system to BW system. Many tRFCs are getting stuck red with message "Error when opening RFC connection". In SM21; we can see below error logs:
RD Q0 I Operating system call recv failed (error no. 104 )
RD Q0 R Call of function GwRead failed, RC: -006
DIA 016 100 ALEREMOTE E0 0 New System Log File Started with Number 18
DIA 016 100 ALEREMOTE R4 9 Communication error, CPIC return code 017, SAP return code 239
DIA 016 100 ALEREMOTE R6 4 > CPI-C function: CMINIT(SAP)
When I checked the RFC connection from SRM to BW system. It is behaving strangely. In 3,4 connections tests. It works fine. Then fails for next two with SAP gateway communication error. The local gateway is running. But sometimes it gives same error in SM58 that Local gateway not running. Below is the gateway trace
Fri Sep 7 04:14:41 2012
***LOG Q0I=> NiIRead: P=3.156.17.106:3300; L=3.156.9.64:36425: recv (104: Connection reset by peer) [nixxi.cpp 5087]
*** ERROR => NiIRead: SiRecv failed for hdl 129/sock 59
(SI_ECONN_BROKEN/104; I4; ST; P=3.156.17.106:3300; L=3.156.9.64:36425) [nixxi.cpp 5087]
*****************************************************************************
*
* LOCATION SAP-Gateway on host ushaplp923la.nbcuni.ge.com / sapgw00
* ERROR read from SAP gateway failed
*
* TIME Fri Sep 7 04:14:41 2012
* RELEASE 720
* COMPONENT SAP-Gateway
* VERSION 2
* RC 239
* MODULE gwdp.c
* LINE 4686
* DETAIL connection to gateway on host 3.156.17.106 / 3300 broken
* COUNTER 8954
*
*****************************************************************************
***LOG Q0R=> GwReadFromRemGw, GwRead ( GwRead-006) [gwdp.c 4713]
The temporary behaviour of problem is strange. The RFC starts working fine always once teh load is finished. No more errors in SM21 as well.
Could you please help me on this issue.
Thank You
Varun
Hello Varun,
Can you add service file entry at windows on both servers
example : sapms<SID> 36xx/txp
and try again.
Thanks
Yogesh
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Yogesh,
Yes; I checked services file entry and they are present on both UNIX servers under /etc/hosts.
Checked note 1519484 recommended by Roman as well; but did not help to find out cause of the issues.
The RFC connect from SRM to BW is sometimes goes successful and sometimes it fails.
Connections from other systems to same BW and from same SRM to other systems are working fine. The issue if only from SRM to BW.
Regards
Varun
Hello Varun,
The entry
LOG Q0I=> NiIRead: P=3.156.17.106:3300; L=3.156.9.64:36425: recv (104: Connection reset by peer) [nixxi.cpp 5087]
*** ERROR => NiIRead: SiRecv failed for hdl 129/sock 59
(SI_ECONN_BROKEN/104; I4; ST; P=3.156.17.106:3300; L=3.156.9.64:36425) [nixxi.cpp 5087]
seems to point to a network issue. Has the network been thoroughly checked?
SAP Note 500235 - Network Diagnosis with NIPING
regards,
Paul
Hi All,
We struggled a lot while going back to again and again to netwrok team and they kept on confirming that everything within network is fine; there is some problem within SAP.
So we did not go through all steps of Paul Power's suggestion note # 500235 and left with few initial steps.
While going through that note again; we ran section 3B of the note bothways. Earlier we had run niping with server on SRM and client on BI system. This time we ran it reverse and everytime it got failed. Which was not the case with normal ping.
So it confirmed us a netwrok issue. Though network team could not resolve that. But we worked into that direction with them further and got to know that the IP address series of SRM(LINUX) system is different from all other systems (On Sun OS). But it had an additional IP address series same as other systems lying unused. We used that IP in niping and it worked fine. Later we added IP to RFCs and it resolved the problem.
We will add it further to DNS to make RFC load balanced once again.
Thanks all for providing your inputs on this issue and helping us resolve.
Regards
Varun
Hi,
I was getting the same error today as well, then instead of using the hostname for the system i was adding, i used the IP address. The next time i tried, it was successful. Hope this is helpful.
regards,
Vimbai
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
We also faced the simlar issue some days back. it was due to max no of gateways exceeded , so we increase the value. Just try this it may work.
Thanks,
Shashi
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi All,
Checked with our server team and network team but they confirmed everything fine. Upon further analysis; found few more details as below. But not able to find any roopt cause of the error
:::::::
Below are few details from the analysis on this issue:
è RFC is using BW user group SERVICE in BW. Which has three app servers. All three are responding while pinging from SRM servers. But the RFC connect from SRM succeeds to same target server some time and fails after few seconds; with same source and target app server
è sapms entries are maintained in services files
è Below are the errors seen in SM21 log which suspects network error. But ping from SRM to BW app servers keeps running without any interruption. Also; it was confirmed as good by server team in ticket I raised for them earlier
DIA 018 100 DDIC SM59 R4 9 Communication error, CPIC return code 017, SAP return code 239
RD Q0 R Call of function GwRead failed, RC: -006
DIA 018 100 DDIC SM59 R6 4 > CPI-C function: CMINIT(SAP)
è In RFC trace there are two different codes. Rc=0 for success and rc=17 for failure
Failure
>>>> [2] UBPCLNT100 : R/3 <ac: 1> K usushapsp693 >>> OPEN
UUID: ab_drvstate create uuid {504EB7B6-62C3-4821-E100-0000039C0940}
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB7B662C34821E1000000039C0940}_0
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB7B662C34821E1000000039C0940}_0
CMINIT(lu:3.156.17.106, tp:sapdp00, gwhost:3.156.17.106, gws:3300) rc = 17
======> CPIC-CALL: 'ThSAPOCMINIT' : cmRc=17 thRc=239
SAP gateway communication error (Is SAP gateway closed?)
ABAP Programm: RSRFCPIN (Transaction: SM59)
User: DDIC (Client: 100)
Destination: UBPCLNT100 (Handle: 2, DtConId: 504EB7B662C34821E1000000039C0940, DtConCnt: 0, ConvId: ,{504EB7B6-62C3-4821-E100-0000039C0940})
EPP RootContextId: 441EA102B4B41ED1BF9CE690224081A5, ConnectionId: 00000000000000000000000000000000, ConnectionCnt: 0
EPP TransactionId: 504EB7B362C34821E1000000039C0940
Error RFCIO_ERROR_SYSERROR in abrfcpic.c : 3110
CPIC-CALL: 'ThSAPOCMINIT' : cmRc=17 thRc=239
SAP gateway communication error (Is SAP gateway closed?)
DEST =UBPCLNT100
HOST =3.156.17.106
SERV =sapdp00
GWHOST =3.156.17.106
GWSERV =3300
>>>> [2] UBPCLNT100 : R/3 <ac: 2> K usushapsp693 >>> CLOSE abrfcio.c 663 ( )
-{504EB7B6-62C3-4821-E100-0000039C0940}
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB7B662C34821E1000000039C0940}_0
==== Delta HO 0, 0 LOG DROPPED
>>>> [2] UBPCLNT100 : R/3 <ac: 3> K usushapsp693 >>> FREE abrfcio.c 664 ( )
-{504EB7B6-62C3-4821-E100-0000039C0940}
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB7B662C34821E1000000039C0940}_0
==== Delta HO 0, 0 LOG DROPPED
SUCCESS
>>>> [2] UBPCLNT100 : R/3 <ac: 4> K usushapsp693 >>> OPEN
UUID: ab_drvstate create uuid {504EB807-62C3-4821-E100-0000039C0940}
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB80762C34821E1000000039C0940}_0
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB80762C34821E1000000039C0940}_0
CMINIT(lu:3.156.17.106, tp:sapdp00, gwhost:3.156.17.106, gws:3300) rc = 0
>>>> [2] UBPCLNT100 : R/3 <ac: 5> K usushapsp693 >>> 75031383 (75031383)
-{504EB807-62C3-4821-E100-0000039C0940}
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB80762C34821E1000000039C0940}_0
Send RFCHEADER: 01/LIT/IEEE/SPACE-U2/4103
Send UNICODE-RFCHEADER: cp:4103/ce:IGNORE/et:3/cs:2/rc:0x00000023
K-CodePages: default=4103, logon_lang=4103, curr_locale=4103, front_locale=4110, default_pcs=1100
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB80762C34821E1000000039C0940}_1
UUID: RfcCallNew send the uuid to the partner {504EB807-62C3-4821-E100-0000039C0940}
==== Delta HO 0, 1 REG. T-ID 00000000000013 pDH 2b5d8775fb38 DH-HO[0,0] LOC 0001 l_o 0 XTYP 040000000000b5e0
==== Delta HO 2, 1 REGH T-ID 00000000000013 pDH 2b5d8775fb38 DH-HO[2,1] LOC 0001 l_o 0 XTYP 040000000000b5e0 (len 0, val 1)
==== Delta HO 2, 1 CONFIRMATION REQUESTED
RFC Info rfctable>: [2] no Compression used
>>>> [2] UBPCLNT100 : R/3 <ac: 6> K 3.156.17.106 >>> WRITE (75031383)
-{504EB807-62C3-4821-E100-0000039C0940}
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=00000000000000000000000000000000}_0-
{root-id=441EA102B4B41ED1BF9CE690224081A5}_{conn-id=504EB80762C34821E1000000039C0940}_1
000000 D9C6C3F0 F0F0F0F0 F0F0F0E3 01010008 ÙÆÃððððððððã....
000010 01010105 04010003 01010103 00040000 ................
000020 061B0103 0106000B 04010003 01030200 ................
000030 00002301 06010500 B4312D31 30312D32 ..#.....´1-101-2
000040 2D343333 342D332D 302D342D 342D352D -4334-3-0-4-4-5-
000050 332E3135 362E392E 36342D36 2D736170 3.156.9.64-6-sap
000060 67773030 2D372D33 2E313631 2E34352E gw00-7-3.161.45.
è In error log; it is not showing any time out error. In gateway parameters; below are a couple of differences noticed. But none of these seem to be directly related to problem.
gw/gw_disconnect =(1800 in SRM) and (3600 in ECC)
gw/cpic_timeout =( 20 in SRM) and (60 in ECC)
gw/reg_no_conn_info = (0 in ECC) and (1 in SRM) { 0- No additional security active. 1- Bypassing security in reginfo & secinfo}
gw/remsh =( /usr/bin/rsh in SRM) and (/bin/ssh in ECC)
overflow_size_limit = (5000000 in SRM) and (20000000 in ECC)
è Few times; for erroneous tRFCs; we get error message that ‘Local gateway not running’. Though it is found running.
Please advise.
Regards
Varun
Hi,
One more finding from the dev_icf file at time of connection failire. It mentiones time out in between.
GwReadSapWpRq2: wp 4 waits for data
DpGetCpicCommIdx: found index 7 via appc_hdr
DpGetCpicCommIdx: found comm entry 7 (tid/uid/mode/conv_id/a_r: 43/2528/0/ /0)
GwInitLocCommAdmEntry: init gw_comm_adm entry 7
GwListRemove: elem 7 not in comm_wait_list
GwListRemove: elem 7 not in comm_lookup_list
GwHdlRqFromSapWp: sap_params->sap_cminit.net_protocol = 0
NiIGetServNo: servicename '3300' = port 3300
GwGetFreeGateway: found free slot 6 after search (1 loops)
NiIGetServNo: servicename '3300' = port 3300
GwHdlRqFromSapWp: try to connect to gw (3.156.17.106 3300)
GwIAsyncConn: connect to gateway 3.156.17.106 / 3300
NiIGetServNo: servicename '3300' = port 3300
NiICreateHandle: hdl 416 state NI_INITIAL_CON
NiIInitSocket: set default settings for new hdl 416/sock 100 (I4; ST)
NiIBlockMode: set blockmode for hdl 416 FALSE
NiIConnectSocket: connection of hdl 416 to 3.156.17.106:3300 in progress (timeout=0)
NiIConnect: hdl 416 took local address 3.156.9.65:40286
NiIConnect: state of hdl 416 NI_CONN_WAIT
GwHdlRqFromSapWp: timeout during GwIAsyncConn
GwSaveExtInfo: 196b28f0 for ext info (496 bytes)
GwListInsert: insert elem 6 into gw_remgw_list (at end)
GwListInsert: 18 elems in gw_remgw_list
GwListRemove: remove elem 6 from gw_free_list
GwListRemove: 982 elems in gw_free_list
GwInsertWaitingAddr: inserted waiting_addr = 3.156.9.65 (1)
GwSaveExtInfo: 196b2af0 for ext info (496 bytes)
GwSelSet: set RW- (2abcfc3e3530) for hdl 416
GwCheckPollValue()
GwCheckPollValue: reset poll value to 3600000 msec
Thu Sep 20 09:13:31 2012
*** found trace flag (2) in appc header ***
GwFiSearchConvId: not found (remove=0)
GwReadSapWpRq: request from sap wp 4
-IN- tid 77 uid 2550 mode 0
-IN- appc_ca_blk 306 len 420 rq_id 32767
-IN- wait_f_answ 1 cmd 0
GwReadSapWpRq2: wp 4 waits for data
DpGetCpicCommIdx: found index 7 via appc_hdr
DpGetCpicCommIdx: found comm entry 7 (tid/uid/mode/conv_id/a_r: 77/2550/0/ /0)
GwInitLocCommAdmEntry: init gw_comm_adm entry 7
GwListRemove: elem 7 not in comm_wait_list
GwListRemove: elem 7 not in comm_lookup_list
GwHdlRqFromSapWp: sap_params->sap_cminit.net_protocol = 0
NiIGetServNo: servicename '3300' = port 3300
GwFoundWaitingAddr: found addr
GwSearch: found gw (3.156.17.106 3300) after search (18 loops) protocol=0
GwSendToRemGw: request for gw 6 (len=420)
GwFiInfoConvId: not found
GwSendToRemGw: send data for conv_ID to
But time out is not 5 seconds neither on source server GW parameters nore on target.
Regards
Varun
For Time out issue I like you to take a look at
Its kind of give you idea..
thanks
Yogesh
Hi Varun,
As per your earlier statements it seems the RFC connection behaves abnormally when the load is running. And also you said the SM59 --> RFC destination --> connection test fails intermittently.
Maybe a throughput issue, in your BW system SM21 do you find any error message related to CPIC or gw ?
What are the parameter values for max gw or CPIC ?
Transaction SMGW -> Goto -> Parameters in both the systems .
Thanks,
Samik
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Samik,
Yes; we find gateway error in corresponding BW system which points to network error.
ALEREMOTE R4 9 Communication error, CPIC return code 020, SAP return code 223
ALEREMOTE R5 A > Conversation ID: 75652025
ALEREMOTE R6 4 > CPI-C function: CMSEND(SAP)
GW parameters in BW System:
gw/max_sleep | 20 |
gw/max_conn | 2000 |
gw/conn_disconnect | 900 |
gw/conn_pending | 60 |
gw/gw_disconnect | 1800 |
gw/req_stack_size | 30 |
gw/max_sockets | 2048 |
gw/cpic_timeout | 20 |
gw/accept_timeout | 60 |
gw/frag_timeout | 120 |
gw/deallocate_timeout | 600 |
gw/reg_timeout | 60 |
gw/reg_lb_level | 1 |
gw/reg_lb_default | 20 |
gw/sec_info | /usr/sap/UBP/D00/data/secinfo |
gw/reg_info | /usr/sap/UBP/D00/data/reginfo |
gw/reg_no_conn_info | 1 |
gw/prxy_info | /usr/sap/UBP/D00/data/prxyinfo |
gw/startup | /usr/sap/UBP/D00/data/gwstartup |
gw/stat | 0 |
gw/max_sys | 2000 |
gw/max_overflow_size | 25000000 |
gw/max_overflow_usage | 20 |
gw/keepalive | 300 |
gw/reg_keepalive | 300 |
gw/compatibility | 0 |
gw/monitor | 1 |
gw/netstat_once | 1 |
gw/close_routes | 120 |
gw/timeout | 0 |
gw/use_udp | 0 |
gw/internal_timeout | 0 |
gw/nitrace | 0 |
gw/local_addr | 0.0.0.0 |
gw/accept_remote_trace_level | 1 |
gw/export_trace_level | 0 |
gw/external_trace_level | 0 |
gw/resolve_timeout | 2000 |
is/use_uds | 1 |
rdisp/max_gateways | 100 |
rdisp/max_comm_entries | 2000 |
rdisp/sna_gateway | usushapsp693 |
rdisp/sna_gw_service | sapgw00 |
gw/rem_start | SSH_SHELL |
gw/remsh | /usr/bin/ssh |
gw/ssh | /usr/bin/ssh |
exe/gwrd | /usr/sap/UBP/SYS/exe/run/gwrd |
snc/enable | 1 |
snc/gssapi_lib | /usr/sap/UBP/SYS/exe/run/libsapcrypto.so |
snc/permit_insecure_start | 1 |
Attributes | |
Release | 720 |
Release no | 7200 |
internal version | 2 |
start time | Sun Aug 19 11:29:40 2012 |
build time | Mar 20 2012 04:27:31 |
build with Unicode | TRUE |
build with Threads | FALSE |
gateway hostname | ushapsp701lc.nbcuni.ge.com |
gateway service | sapgw00 |
overflow_size_limit | 5000000 |
overflow_use | 0 |
trace level | 1 |
trace level external programs | 0 |
PID Gateway Reader | 9196 |
GW parameters in SRM system:
gw/max_sleep | 20 |
gw/max_conn | 2000 |
gw/conn_disconnect | 900 |
gw/conn_pending | 60 |
gw/gw_disconnect | 1800 |
gw/req_stack_size | 30 |
gw/max_sockets | 2048 |
gw/cpic_timeout | 20 |
gw/accept_timeout | 60 |
gw/frag_timeout | 120 |
gw/deallocate_timeout | 600 |
gw/reg_timeout | 60 |
gw/reg_lb_level | 1 |
gw/reg_lb_default | 20 |
gw/sec_info | /usr/sap/UPP/D00/data/secinfo |
gw/reg_info | /usr/sap/UPP/D00/data/reginfo |
gw/reg_no_conn_info | 1 |
gw/prxy_info | /usr/sap/UPP/D00/data/prxyinfo |
gw/startup | /usr/sap/UPP/D00/data/gwstartup |
gw/stat | 0 |
gw/max_sys | 2000 |
gw/max_overflow_size | 25000000 |
gw/max_overflow_usage | 20 |
gw/keepalive | 300 |
gw/reg_keepalive | 300 |
gw/compatibility | 0 |
gw/monitor | 1 |
gw/netstat_once | 1 |
gw/close_routes | 120 |
gw/timeout | 0 |
gw/use_udp | 0 |
gw/internal_timeout | 0 |
gw/nitrace | 0 |
gw/local_addr | 0.0.0.0 |
gw/accept_remote_trace_level | 1 |
gw/export_trace_level | 0 |
gw/external_trace_level | 0 |
gw/resolve_timeout | 2000 |
is/use_uds | 1 |
rdisp/max_gateways | 1000 |
rdisp/max_comm_entries | 2000 |
gw/rem_start | SSH_SHELL |
gw/remsh | /usr/bin/rsh |
gw/ssh | /usr/bin/ssh |
exe/gwrd | /usr/sap/UPP/D00/exe/gwrd |
snc/enable | 0 |
snc/permit_insecure_start | 0 |
Attributes | |
Release | 720 |
Release no | 7200 |
internal version | 2 |
start time | Tue Aug 21 05:24:51 2012 |
build time | Mar 20 2012 00:49:23 |
build with Unicode | TRUE |
build with Threads | FALSE |
gateway hostname | ushaplp923ld.nbcuni.ge.com |
gateway service | sapgw00 |
overflow_size_limit | 5000000 |
overflow_use | 0 |
trace level | 1 |
trace level external programs | 0 |
PID Gateway Reader | 25291 |
SRM system is on LINUX and BW is on Sun OS> We are not getting any max number of gateway connetions like issue
Thanks,
Varun
Try to check Note 1519484 - How to analyze network disconnections shown in system log (BC transaction SM21)
This behavior may be due to many reasons: network, firewall activity, OS settings, etc. This issue required more investigation. At first read carfully note above
Regards
Roman
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
85 | |
23 | |
11 | |
9 | |
8 | |
5 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.