File content is 'jumbled' or mixed up

[mahesh_madhavan]

Hello,

We have a program which creates text files.

The content is output of table AENR and the related characteristic values of a change number. The file has over half a million records.

The issue is:

• The output gets mixed up in the file sometimes.
• The value in one cell appears in a different column or in a different row.
• The issue occurs intermittently and there is no issue with the final output table when we do a debug and the session creates the file with perfect output.

Has anybody here experience similar issue?

Regards,

Mahesh

[Sandra_Rossi]

Sorry, but your question is missing the most important information: how does your custom program work? What you say about the debug makes me think that you write in this file in parallel sessions. This won't work : every write to the dataset will do it "randomly", the operating system or the file system will receive blocks of data to write, and one block may arrive at the middle of another block. You can't write parallely. The only way to do it is to write to separate files (one per ABAP thread), and at the end you merge the files into one (of course, performance decreases a little bit, but there's no choice).

[mahesh_madhavan]

Hello Sandra,

Really sorry for my late response!

• This file is not written in parallel processing
• In a single run we create a file
• However sometimes this file has its contents jumble up
• To investigate the issue, we ran the program in debug mode. However the final internal table which holds this data which is to be written to the file had the content perfectly. It created the file with perfect content. Hence whenever we debug, the issue is not occurring.
• It occurs when the program is run in background mode.

Hope I answered your queries. Thanks for your reply.

Regards,

Max

[matt]

It doesn't answer the query as you still haven't said how you generate the files.

[horst_keller]

You don't say, if it is a file on the application server or on the presentation server ...

[mahesh_madhavan]

Hello Horst,

Sorry! It is written to application server.

Something I missed to add, after writing the files to application server, it is zipped to a tar.gz file - since there are multiple files generated by the program in huge size(it is only 1 file out of all the 10 files that has an issue)

1. Rename file to .dat

* need to change the permission of the file
  CONCATENATE c_chmod_777 gv_file INTO gv_permission SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND' FIELD gv_permission
                ID 'TAB'     FIELD TABL-*SYS*.
  IF sy-subrc <> 0.                                            "#35712
       MESSAGE I000 WITH text-E11 sy-subrc lt_files-name(50)   "Change permission FAILED, subrc =
                                           lt_files-name+50(25).
  ENDIF.


* rename .dat
  CONCATENATE gv_sap_source_path gv_sap_source_file c_dash gv_date c_dot 'txt' INTO gv_renamed.
  CONCATENATE 'mv' gv_file gv_renamed INTO gv_full_command  SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND'         FIELD gv_full_command     "rename     - /process/*.dat -->  /process/*_yyyymmdd.txt
                ID 'TAB'             FIELD TABL-*SYS*.
  IF sy-subrc <> 0.                                            "#35712
     MESSAGE I000 WITH text-E12 sy-subrc lt_files-name(50)     "Rename FAILED, subrc =
                                         lt_files-name+50(25).
  ENDIF.<br>

2. Compress all the 10 files into a compressed file

 DATA: lv_message(100). "#35712


  CONCATENATE 'SAP_REPORTS' c_dash gv_date c_dot 'tar' INTO lv_message. "#35712


* 1- zip files
  CONCATENATE p_process_dir 'SAP_REPORTS' c_dash gv_date c_dot 'tar' INTO gv_file.
  CONCATENATE 'tar' '-cf' gv_file '-C' p_process_dir gv_zip_files INTO gv_full_command  SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND'            FIELD gv_full_command                           "create     - /process/SAP_REPORTS_yyyymmdd.tar
                ID 'TAB'                FIELD TABL-*SYS*.
  IF sy-subrc <> 0.                                        "#35712
     MESSAGE E000 WITH text-E13 sy-subrc lv_message.       "Command tar FAILED, subrc =
  ENDIF.
  CONCATENATE 'gzip'  gv_file                                     INTO gv_full_command  SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND'            FIELD gv_full_command                           "create     - /process/SAP_REPORTS_yyyymmdd.tar.gz
                   ID 'TAB'             FIELD TABL-*SYS*.                               "delete     - /process/SAP_REPORTS_yyyymmdd.tar
  IF sy-subrc <> 0.                                        "#35712
     MESSAGE E000 WITH text-E14 sy-subrc lv_message.       "Command gzip FAILED, subrc =
  ENDIF.


* Need to put a wait 5 min., because we wont to sure that ZIP files is finish, before you move the zip files
  WAIT UP TO 300 SECONDS.


  CONCATENATE lv_message  '.gz'      INTO lv_message.      "#35712


* 2- need to change the permission of the file
  CONCATENATE gv_file  '.gz'      INTO gv_file.
  CONCATENATE c_chmod_744 gv_file INTO gv_permission SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND' FIELD gv_permission                                        "permission - /process/SAP_REPORTS_yyyymmdd.tar.gz
                ID 'TAB'     FIELD TABL-*SYS*.
  IF sy-subrc <> 0.                                        "#35712
     MESSAGE E000 WITH text-E11 sy-subrc lv_message .      "Change permission FAILED, subrc =
  ENDIF.


* 3- move .tar.gz to /reports
  CONCATENATE p_reports_dir 'SAP_REPORTS' c_dash gv_date c_dot 'tar.gz' INTO gv_renamed.
  CONCATENATE 'mv' gv_file gv_renamed INTO gv_full_command  SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND'         FIELD gv_full_command                              "move       - /reports/SAP_REPORTS_yyyymmdd.tar.gz
                ID 'TAB'             FIELD TABL-*SYS*.
  IF sy-subrc <> 0.                                        "#35712
     MESSAGE E000 WITH text-E15 sy-subrc lv_message .      "Command move FAILED, subrc =
  ENDIF.


<br>

Only one file has got it contents jumbled up.

Thanks,

Mahesh

[Sandra_Rossi]

No, your issue is not about the rename, not about the compression, not about the move, it's only an issue when you generate the files.

So, please describe the way you generate 10 files. Why do you generate 10 files, and not 1 ? I still assume (as I already said) that you are generating these files parallely for performance reason. So, based on the information you have give, the most probable reason of the issue is that 2 processes are writing to the same file at the same time.

[matt]

Just as a comment:

 CONCATENATE c_chmod_777 gv_file INTO gv_permission SEPARATED BY space.
  CALL 'SYSTEM' ID 'COMMAND' FIELD gv_permission
                ID 'TAB'     FIELD TABL-*SYS*.<br>

is a recipe for code injection. You must sanitise gv_file. Consider if gv_file contained

myfile.txt;rm -rf ../../../<br>

Now you've potentially deleted everything that the sap linux account has access to, from three folders up from you default folder.

edit: following on from Horst's link below - don't use CALL 'SYSTEM' at all. There are safer alternative.

[horst_keller]

https://blogs.sap.com/2013/03/06/a-word-about-call-system/

[horst_keller]

... or the contents of the internal table is already "jumbled up".

[former_member182550]

Also - how are you looking at the resultant file ?

If it's a tab delimited file then the position of the next 'column' can be dependant on the length of the data in the previous rows visually, but in the file it is in the right place.