cancel
Showing results for 
Search instead for 
Did you mean: 

SWPM - Parallel Ex-/Import - Slow FTP

Trompetter
Explorer
0 Kudos

Dear all,

we are using SWPM for parallel export and import our SAP system. Transfer to remote system is done by FTP within SWPM.

When we use FileZilla for transferring one file to the remote FTP server our 1 GBit network adapter is used by around 20%.

When we use SWPM and one file transferred it uses only around 2% of the 1 GBit network adapter. So we have to use 50 parallel processes and enough completely exported files to get a high transfer rate. Please find attached screenshot with transfer rates over the whole process.

Your suggestions how we can speed up the transfer rate are highly appreciated.

Best regards

Peter

Accepted Solutions (0)

Answers (1)

Answers (1)

Former Member
0 Kudos

Hello Peter,

Be it native ftp or filezilla I wouldn't expect a real difference with the throughput rate.

If I look at your screen shot it's about after 3 hours of your export running that you seem to hit the bigger tables and then your network utilization increases due to bigger files being transferred.

If you want to optimize your export then I would suggest you run MIGTIME which will show you the packages that took the longest time to export and the start-end time of each package and the sizes.

From the info above, for your next run you can create a orderBy file where you determine which packages will be unloaded first (longest runners - largest dump fles). Initially you wont see much being transferred over the network as the export is still running on the long runners. Once the long runners write out the first of their 1GB files then will see a sustained usage of your network bandwidth.

Over and above that, there are many ways to optimize OSDB such as package/table splitting etc .... but that's another subject.

Just be careful to not go overboard with the number of parallel processes. Watch you CPU utilization during the export/import and ramp up accordingly.

KR,

Amerjit

Trompetter
Explorer
0 Kudos

Dear Amerjit,

thank you for your suggestions. We already use table splitting to optimize the whole procedure.

If SWPM transfers one big file of around 1 GB network utilization is around 2% if nothing else is transferred. Same file to the same server with Filezilla is around 20% network utilizations.

Our sending server has around 90 CPU cores, so 50 processes in parallel should be ok.

Best regards

Peter

Former Member
0 Kudos

Hi Peter,

1. 90 cores (nice) ... Then 50 processes is fine 🙂

2. What's your o/s ?

3. Have you tried a manual ftp outside of SWPM. I'd do a manual ftp on the o/s using *exactly* the same file you tested your transfer with using filezilla. Monitor your bandwidth utilization when using native ftp and filezilla. FTP is so basic I'd be surprised (not shocked) if there was such a large difference by using one tool over an other.

Now thinking about it, what filezilla could be doing is breaking your file into smaller packages and then sending the NN packages in parallel. You could check that by running netstat on your source machine during the transfer to see the number of open sockets towards your target machine.

KR,

Amerjit

Trompetter
Explorer
0 Kudos

Dear Amerjit,

1. yes, it´s our productive PAS 🙂

2. Windows 2008 R2 Enterprise

3. I used FileZilla and the normal Windows FTP and both are fast. In the meanwhile I also think that at least the FileZilla "optimizes" the transfer by splitting up the files and using more channels. Maybe SWPM uses one single channel to transfer the data.

I already opened a case at SAP but they asked to update the Kernel or at least R3load and the Database Library but that´s not possible at the moment and I also think it would not help. Everything works fine but the FTP transfer.

Best regards

Peter

Former Member
0 Kudos

Hi Peter,

SWPM doesn't have it's own ftp (as far as I know anyway). It just makes a call to the default ftp client installed on the server and your basic ftp client is just that .... basic ...... No threading, splitting etc ....

As a general rule, I always update the R3* tools before I do a OSDB. They are independent of the kernel anyway (keeping on the same release of course).

From what you're describing it wouldn't help as you are clearly stating that you problem is shifting the data with FTP.

I'd still ask you to consider running MIGTIME, analyse your export packages and then put the longest running/biggest export at the beginning of your orderBy list.

When you transfer a file with built in ftp, what throughput are you getting MB/s ?

Just one final thing, what is the value of ftpJobNum in your export_monitor.properties file ? You can increase the number of parallel ftp processes with that parameter.

I hope all the above has helped a bit.

KR,

Amerjit