Dictionary Issue - Base Data Cleanse

[Former Member]

Hello Experts,

Using Data Services 12.2 .In my dictionary there are 590 classifications are there(I think this is the maximum limit after that it giving error unable to allocate index ,indexes are over).my source contains 16500 records ,i ran this job on server with 4GB RAM ,for loading 10000 erecords it took 6 hours i forcibly killed the job after that. i tried with run as seperate process but no luck.Please let me know how to tune this datacleanse transform to improve the performance.For testing purpose, i am using query transformation after that will be using BDC transformation then output table .

Please give your inputs?

Will it be possible by splitting dictionary and doing so the performance takes a big hit?

Thanks

[Former Member · Accepted solution]

Hi Felix,

Could you provide some more information so I can understand the problem better.

1. Are you sure you have 590 "classifications" and not some other dictionary attribute like Primary_Entry?

2. Are you using a shipped cleansing package or using your own custom dictionary? If you are using a custom dictionary, can you provide an idea of the size of the dictionary in terms of number of primary entries? Just a ball park number to get an idea would be sufficient. Would be best if you can export your dictionary and pass it along so I could take a look.

3. What kind of RDBMS is your dictionary repository based?

Based on your response here I may have some follow up questions. In the meantime, can you also provide some description of what you are trying to achieve in your dataflow, is it some Name/Title/Firm processing or custom data processing?

To answer some of your queries, you can't really split the dictionary and looking at the number of records you are handling, it shouldn't take that long a time. There might be something else in the setup that we might need to look into.

Thanks,

Sarthak

[venkataramana_paidi · Accepted solution]

Hi Sarthak,

Here am giving the brief description of the my dictionary.

I have the input of material description. am parsing the material description into different attributes like size,color,width,length,outer diameter, speed etc....In these attributes some are static I mean we enter the values into dictionary for example color. For this color we enter the color values into dictionary as primary entries. I loaded this type of static attributes into one dictionary. Till yet I entered more than 12000 entries its working fine.

In the attributes some of the attributes are dynamic. For example take speed of the motor 1000 rpm. in this value unit of measure rpm is constant but speed value may changes.Thats why each dynamic attribute values classified into patternuomid here pattern means numeric or alpha numeric values,uom means cm,mm,rpm,....,id means for temperature F or C,for inside diameter ID like that.

Here I give one example

In the description inside diameter is there. This attribute will come in this possible ways

10 IN ID

10INID

10IN ID

10 INID

10 ID

10ID

Here am using break on white space using in my base data cleanse transformation.Thats why I need to write some times 10 classifications for each attribute. In this way classifications are increasing .

2) Am using mysql database for dictionary repository.

Please look into this one give me your valuable suggestions.

Thanks,

Ramana.

[Former Member · Accepted solution]

Hi Ramana,

Thanks for the detailed description of your project.

Here is what I think is happening:

1. You are trying to bulkload a LOT of classifications - using the bulkload too in designer.And you get an error saying "App failed to assign classification index". Is that correct?

2. If you go to the Dictionary Menu (in DS Designer), select your dictionary and just hit the "Search" button, you get nothing back. Could you verify this as well?

If the above 2 are true, then the issue you are seeing is because of the dictionary not loading correctly. At this point we will need to evaluate your approach in the project. It is not typical to have these many classifications in a project, usually the same can be achieved with significantly lesser number of classifications.

Would it be possible for you to send/attach your rule file? This will help me evaluate your usage of these classifications and suggest an alternate approach.

Thanks,

Sarthak

[Former Member · Accepted solution]

Just wanted to add that if you manually entered the dictionary entries, then doing a Search on them will probably return your primary entries. I would be curious how you are loading your primary entries and classifications - Manually through the Dictionary menu or via Bulkload.

[Former Member · Accepted solution]

Hello,

While we are working through this issue, it might be easier if we channel this through our customer support. For this, could you create a SAP message (under BOJ-EIM-DS).

If you are unsure about how to create a support SAP Message, please contact me directly (see address under my profile - https://forums.sdn.sap.com/profile.jspa?userID=3850954) and I can work with you to get there.

Thanks,

Sarthak

[Former Member]

As a follow up to anyone else who might be running into the same issue, the cause of the error "Unable to allocate index..." is due to the limit on the number of classifications allowed in a dictionary. Once that limit is hit, any attempt to add new classifications will result in that message.

Also regarding the performance hit in this case, it is related to heavy usage of Pattern based classifications. Each token in the record is matched against every pattern based classification(PBC) causing a performance hit and it is not a recommended best practice to have a huge number of PBC's. PBC's are mainly in place to handle special cases such as parsing $10,000 and similar notations. A better approach would be to leverage the dictionary and use custom classifications to optimize the performance.

[Former Member]

Thank you.

I guess we will probably speak to SAP on this.