Checking count of raw data loded in to datahub and published to hybris

[Former Member]

I am using CSV adapter for data import to data hub. I would like to know how many raw data got loaded to data hub and publish to hybris. I can see the count of raw data items from log. I am loading 5k raw data in to data hub and I would like to know whether all 5k rows got published and if not which all raw data loaded in to data hub is not published to hybris with error details. Is this a customization in the hybris adapter or any OOB feature is available ??

[former_member224482 · Accepted solution]

Here is how you can find out using the existing REST API.

Total amount of canonical item updated during a single composition :

 http://localhost:8080/datahub-webapp/v1/pools/GLOBAL/compositions/1
 <compositionActionData>
   <actionId>1</actionId>
   <type>COMPOSITION</type>
   <startTime>2015-10-29T11:07:02.455-04:00</startTime>
   <endTime>2015-10-29T11:07:02.856-04:00</endTime>
   <status>SUCCESS</status>
   <poolName>GLOBAL</poolName>
   <count>2</count>
 </compositionActionData>

The count 2 of the composition is the number of canonical item updated by the composition.

Checking and tracing the publications errors back to the raw items :

 http://localhost:8080/datahub-webapp/v1/pools/GLOBAL/publications/1
 <publicationActionData>
   <actionId>1</actionId>
   <type>PUBLICATION</type>
   <startTime>2015-10-29T11:07:03.831-04:00</startTime>
   <endTime>2015-10-29T11:07:05.778-04:00</endTime>
   <status>FAILURE</status>
   <poolName>GLOBAL</poolName>
   <targetSystemPublications>
     <publicationId>1</publicationId>
     <startTime>2015-10-29T11:07:04.002-04:00</startTime>
     <endTime>2015-10-29T11:07:05.755-04:00</endTime>
     <status>FAILURE</status>
     <targetSystemName>HybrisCore</targetSystemName>
     <actionId>1</actionId>
     <poolName>GLOBAL</poolName>
     <numberOfErrors>1</numberOfErrors>
   </targetSystemPublications>
 </publicationActionData>

The publication will display the targetSystemPublicationId 1 which is in error.

 http://localhost:8080/datahub-webapp/v1/pools/GLOBAL/items/CanonicalBaseProduct.xml
 <itemDatas>
   <itemData>
     <type>CanonicalBaseProduct</type>
     <dataPool>GLOBAL</dataPool>
     <id>2</id>
     <attribute>
       <name>compositionAction</name>
       <values> <value xsi:type="xs:string">1</value> </values>
     </attribute>
     <attribute>
       <name>rawItems</name>
       <values>
         <value xsi:type="xs:long">1</value>
         <value xsi:type="xs:long">2</value>
       </values>
     </attribute>
     <attribute>
       <name>publicationStatuses</name>
       <values>
         <value xsi:type="canonicalItemPublicationStatusData">
         <targetSystemPublicationId>1</targetSystemPublicationId>
         <targetSystemName>HybrisCore</targetSystemName>
         <type>CanonicalBaseProduct</type>
         <status>FAILURE</status>
         <statusDetail>INCOMPLETE</statusDetail>
       </value>
       </values>
     </attribute>
     ...
   </itemData>
 </itemDatas>

The canonical item used in targetSystemPublicationId 1 was last update by the raw items id 1 and 2.

 http://localhost:8080/datahub-webapp/v1/pools/GLOBAL/items/RawMATMAS.xml

Allow you to read the raw item data. Look for id 1 and 2.

[Former Member · Accepted solution]

Thank you for the clarification. Can we use back office API to get the status of how many raw got loaded and how many got published ? I hope using REST API ,we will get the status as the entire feed got failed or success If only one raw data /record (for eg: 501th record in 1000 record feed) got an issue with the data and import doesn't happened for rest of the records, Is there any way to find out the error details with the record info ?

[rafal_gdowski · Accepted solution]

There is a backoffice API that will show your all Data Loading actions that have failed, and the count of the failed items. There is no validation on individual raw items, so as a result we do not propagate errors, just failures of the entire data loading action. Individual errors of raw items (i.e. missing integration key, or isoCodes) will surface during the composition action for the pool where the data was loaded).

 http://localhost:9797/datahub-webapp/v1/data-feeds/data-loads