Large data through JDBC adapters - restart

[Former Member]

We have a big file with more than 20000 records, that needs to be loaded onto database table.

Should we split the files into multiple messages (1000 records each) and try to insert into Database table? Or Should we insert all record in one stretch.

If we follow the first approach, how do we go about restarting the message if there are any insert failures?

[agasthuri_doss · Accepted solution]

Hi,

> Should we split the files into multiple messages (1000 records each) and try to insert into Database table? Or Should we insert all record in one stretch.

1) From the Source side you can split and send

2) Multiple messages is advicable

Shall we know the Source ?

Regards

Agasthuri Doss

[Former Member · Accepted solution]

Source is going to be a file. And we are looking at sending this feed weekly or daily.

[Former Member · Accepted solution]

Hi Kris,

i think number of records doesn't mean to XI... only the size of file.

what would be the size of your text file with 20000 records?

you have to check network traffic too while doing mass uploads...

Regards,

Sukarna

[justin_santhanam · Accepted solution]

Kris,

If there is possibility then I would suggest you to go for Splitting the file from the source. Then send it accordingly. Also, note that when u are trying to do insert 1000 records, if it's failed at 1000 record, then everything else will rollback. I would suggest you to use Synchronous in the Inbound side.

raj.

[Former Member · Accepted solution]

Raj,

We are probably looking at 20M or 30M size files every day. SAP program is extracting this to flat file (so we are looking at file conversion). My requirement is to load all the records into DB table? What is the best way to go about doing this?

[justin_santhanam · Accepted solution]

Kris,

Since you are using Content conversion, just set Recordset per message for ex 1000 and do the rest of the stuff. Do u want to see the error in SXMB_MONI if it fails? If yes then make it as Synchronous Inbound.

The best process as mentioned above

- Split the file.

- Create Asynh O/B and Synch I/B

If you finalize the above steps then u can go ahead with mapping.

raj.

[justin_santhanam]

Kris,

Are u going to use BPM? Is it one time Mass upload or this data is expected all the time?

raj.