JCo IDoc program outofmemory error

[Former Member]

Hi, I'm a new one in SDN and this is my first topic.

Now i use Jco-idoc jar to do some programs, and these programs are very sample: recieve IDOC requests,download to flat file and then do the other things with these files.

When i send a small IDOC request to this program manually (use WE19), all things look very well. But when i try to send a big one ,the program throw exception as below and the program crashed:

<b>Exception in thread "JCO.ServerThread-1" java.lang.OutOfMemoryError: requested 196370116 bytes for jchar in D:/BUILD_AREA/jdk142-update/ws/fcs/hotspot\src\share\vm\prims\jni.cpp. Out of swap space?</b>

I try to increase the JVM with "-Xms1024M -Xmx1024M -XX:NewSize=256M -XX:MaxNewSize=256M -XX:MaxPermSize=256M", but this error as same as before!

Is there buffer space limitation for JNI interface in sapjcorfc.dll? I guess. And the other question confused me. When i use EDI write this IDOC request to SAP local file, the file size only has about 20M and 92363 rows, why in my program recieve "<b>196370116 bytes</b>"?

I work in winxp, JSDK is j2se 1.4.2_08-b03, JCO version is 2.1.5, SAP Java Base IDoc Class Library version is 1.0.2, and SAP Java Connector IDoc Class Library version is 1.0.3.

Thank you!

Regards,

Liu Hui

[Former Member · Accepted solution]

Hi Liu Hui,

not sure if the JCo IDoc implementation adds lots of overhead, but depending on the used IDoc there might be some (did you try to profile it and check how much memory your smaller objects need?).

If you're looking for flat files you might want to try substituting the IDoc server by a simple RFC server program that implements for example IDOC_INBOUND_ASYNCHRONOUS (or whichever function module you're using to send the data). This should be more efficient since at least the SDATA part is transferred as a String, which seems ok, since you're talking of flat files.

Did you also just try running the server with parameter "-Xms1024M -Xms1024M"? Seems like you're making the Eden and Perm space quite large, which leaves less for the heap. I'd give it a shot without the other additions.

Best wishes, harald

[Former Member · Accepted solution]

Harald,

Thanks your reply very much.

I searched the SAP notes, and found some infomation in note 634689, section "Memory problems in native (JNI) part" as below:

*******************************************************

Some java packages use methods implemented in C/C++ via Java Native Interface (JNI), for example Java Connector (JCO). The native coding can also allocate memory via malloc(). In some cases JCO catches the allocation errors in the C-coding and throws an OutOfMemoryError. This kind of OutOfMemoryError can be identified through its message text "Out of memory, last request was %d bytes". If you see such an error you can be sure it comes from the JNI part of JCO.

Similar to the above described thread problem the increasing of -Xmx value (i.e. memory reserved for Java) will decrease the amount of memory available to JNI. In this case one should rather decrease -Xmx value a bit. Unhandled allocation failures in the native coding can also cause process crashes with "ACCESS_VIOLATION". In such a case usually an HotSpot Error occurs. Please always check the server folder for presence of hs_err, javacore, java_core* and core files.

*******************************************************

But whatever i adjust the memory parameter over -Xmx value, i haven't recieved more than 20M (100,000 rows* 200 byte per row) idoc. It's very disappointed, because i have some existed ABAP programs ,which produce very big internal tables and send them to idoc with MASTER_IDOC_DISTRIBUTE. I try to change the partner profile and port to transfer the idoc to external RFC server, so i use jco-idoc package to download these idocs to XML files.

For your suggestions, invoke some bapi functions to get. Would you please say in detail (which function module,etc )or give me some examples.

Thanks and regards,

Liu Hui

[Former Member · Accepted solution]

Hi Liu Hui,

thanks for listing the OSS note, that is interesting.

I don't have time to do some profiling, but it still seems odd that the Object overhead in Java is so huge that a 20MB IDoc get's bloated into such a large Java Object.

If that is true, the alternative I was thinking of was using a normal RFC server program and implementing the RFC module IDOC_INBOUND_ASYNCHRONOUS in the server. This function module will be used by SAP to send the data to your server program (sounds at least like it, from the scenario you're describing).

It is then straightforward to convert the parameters from the function module into an IDoc flat file as used by SAP.

Best wishes, harald

[Former Member · Accepted solution]

Thank you, harald. sorry to reply so late, because Oct,1 is the National day in China and it is meaning a long vacation.

I try to utilize the sapjco.jar to recieve IDOC structures of FM IDOC_INBOUND_ASYNCHRONOUS, but the result is the same as before.

========================================

Exception in thread "JCO.ServerThread-1" java.lang.OutOfMemoryError: requested <b>381395896</b> bytes

for jchar in C:/BUILD_AREA/jdk1.5.0_02/hotspot\src\share\vm\prims\jni.cpp. Out of swap space?

========================================

Then the java application crashed. But, i only send data package about <b>180,000 rows and 35M bytes</b> in sap r/3.

Whether this dues to the JCo version (I use sapjco-ntintel-2.1.5 )or my java command line parameter error(I use -Xmx512M), I guess. Have you any ideas?

Liuhui

[Former Member · Accepted solution]

Hi Liu Hiu,

if I had some more time I'd do some profiling to check the overhead the java layer adds. And then in the end we're still left with the JNI that does the marshalling...

Anyhow, let's make some simplified assumptions here: You say you have 180,000 rows (= IDoc segments). Looking at IDOC_INBOUND_ASYNCHRONOUS the function module's parameter IDOC_DATA_REC_40 for transferring the segments is a structure that needs at least 1,063Bytes. So 1,063Bytes times 180,000 rows makes around 182MB of data (not counting any overhead for the structures itself).

How do you get to the 35MB figure? Are you writing the data to a file and checking filesize? I think SAP truncates the trailing spaces on the IDoc files (ususally the structures occupy way less than the 1,000B of the SDATA field). I'd expect that with the underlying RFC layer the SDATA field is really 1,000B per segment.

Unless you're interested in the general problem (where does the limit reside and how to tweak it), the question might be why you would want to send such a huge IDoc (or IDoc packet)? A single IDoc with 180,000 segments is not desirable in SAP, so I doubt you have that case. If you send multiple IDocs as packets, you should try sending them in smaller packets.

Best wishes, harald

[Former Member · Accepted solution]

Hi, harald

I am not making the general test of what size sap can send thru IDOC. For we must extract some data from sap system of a bank in our project, and the size of data is too large(about over 5GB per datasource) to send to external system synchronally. we decide to build a JCo application so as to recieve the idoc request from sap EDI/ALE. we are aware of the large size of data, so we have to split them into many small packages, for example, 180,000 rows times 157byte/row is around 30MB per package, if we download to local file with EDI.

your words make me clear why the overhead of IDOC so big. As you say, the EDI_DD40->SDATA will be filled with spaces upto to 1000 by itself, while we try to send a small segment that less than 1000 bytes.Since the case, we have to build a ABAP to tranlate the EDI_DD40->SDATA to string, and transfer it to our java service.

thanks again ,harald.

Liuhui

[Former Member]

Is there someone can respond? Thanks your help.

Liu hui