Skip to Content

How to process a csv-file line by line? -> create OData payload for every csv-row

Dear Community,

I'd like to set up an Iflow, that processes a csv file with 100 customer records, validate it and loads it as payload to hybris marketing OData interface.

I use the filter-element to filter out odd customer records e.g. with missing values.

I also use a XML validation (after mapping) to be sure, that all values are valid. Sometimes one customer record out of 100 records is invalid. In that case the validator throws an error and no record has been processed - I think because all records are grouped in one root-xml-element:

<List>

<Cust>
<name>N1</name>
…
</Cust>

<Cust>
<name>N2</name>
…
</Cust>

<Cust>
<name>invalid value</name>
…
</Cust>
</List>

My questions:

How can I modify my Iflow, so that 99 records are processed and only the odd one is ignored?

I think the Iflow should create a own xml payload for every single customer record? Is this possible or do you have any better suggestions?

Thank you!
Best regards,
Peter

iflow.png (18.4 kB)
Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

4 Answers

  • Aug 22, 2017 at 05:14 PM

    Hello Peter,

    it would be great if you can tell us about validation you want to do( May be I can help you with filter condition )Most of the validations can be done in Content Filter only.

    Below blog of mine might help you to find out use cases for Content Filter.

    https://blogs.sap.com/2017/06/01/sap-cloud-platform-integration-content-filter-in-detail/

    Regards,

    Sriprasad Shivaram Bhat

    Add comment
    10|10000 characters needed characters exceeded

  • Aug 22, 2017 at 06:28 PM

    Hi Peter,

    you might be able to solve this in the Message Mapping. Just looking on the example payload in your question, you could exclude invalid values in the mapping. This works when you know why certain data cannot be written, e.g. invalid characters.
    Since the XML validator also throws an error when the payload is invalid, I assume that this is something you can fix before sending the message to the Marketing system.

    Otherwise follow Sriprasad's blog post to see if the Content Filter is a valid option. I don't recommend splitting the message, sending seperate message. This works for 100 messages but you might have a lot more in a productive system.
    In case of an error what information does the receiver system provide?

    br,

    Johannes

    Add comment
    10|10000 characters needed characters exceeded

  • Aug 22, 2017 at 06:28 PM

    Hi Peter,

    you might be able to solve this in the Message Mapping. Just looking on the example payload in your question, you could exclude invalid values in the mapping. This works when you know why certain data cannot be written, e.g. invalid characters.
    Since the XML validator also throws an error when the payload is invalid, I assume that this is something you can fix before sending the message to the Marketing system.

    Otherwise follow Sriprasad's blog post to see if the Content Filter is a valid option. I don't recommend splitting the message, sending seperate message. This works for 100 messages but you might have a lot more in a productive system.
    In case of an error what information does the receiver system provide?

    br,

    Johannes

    Add comment
    10|10000 characters needed characters exceeded

  • Aug 22, 2017 at 06:29 PM

    Hello Sriprasad,

    I do several validations - currently I validate ervery field (aprox. 30 fields) against the xsd (generated from OData EDMX).
    I check if:

    • all mandatory fields contain values
    • the length is within the max. lenght of each field
    • the e-mail fulfills a regex-rule

    Do I need multiple content filters?

    Thank you!
    Best
    Peter

    Add comment
    10|10000 characters needed characters exceeded

    • Hello Peter,

      I have tried with some random test data and test condition ( My own:) and its possible to have all the above 3 cases in Content filter.

      Input XML:

      <root>
      	<record>
      		<field1>query</field1>
      		<field2>123</field2>
      		<field3>Sriprasad</field3>
      		<field4>Bhat</field4>
      	</record>
      	<record>
      		<field1>qery</field1>
      		<field2>124</field2>
      		<field3>sri</field3>
      		<field4>Bhat</field4>
      	</record>
      	<record>
      		<field1>query</field1>
      		<field2>1</field2>
      		<field3>k</field3>
      		<field4>Bhat</field4>
      	</record>
      	<record>
      		<field1>query</field1>
      		<field2>14</field2>
      		<field3>k</field3>
      		<field4></field4>
      	</record>
      </root>
      

      Content Filter XPath Condition:

      root/record[(matches(field1/text(), '[a-z]{5}')) and ((string-length(field2))>=2) and (field4/text() != '') ]

      Output:

      <record>
      	<field1>query</field1>
      	<field2>123</field2>
      	<field3>Sriprasad</field3>
      	<field4>Bhat</field4>
      </record>

      If you are having more fields you just need to be very careful while creating condition and make sure you are gropuing the conditions properly.

      Hope above helps you.

      Regards,

      Sriprasad Shivaram Bhat