Creating Index - Text Analysis

[Former Member]

Hi ,

Just started exploring on the text analysis capabilities in SAP HANA and came across creating indexes to enable text analysis.

I have a table with customer details which I want use for text analysis, so created a index on the column on which I want to perform the analysis.

However after I have created the index when I try to preview the data in the $TA table I do not see any data.

I imported the Delivery Unit as I read some posts mentioning about it. But nothing worked!

Any help would be grateful.

Thanks,

Indu

[former_member182114]

Hi Indu,

Please share the create index statment you used to create this index.

See also if there's no queue in background:

SELECT * FROM SYS.M_FULLTEXT_QUEUES;

Look for your table, if is there, and compare the columns TOTAL_DOCUMENT_COUNT versus INDEXED_DOCUMENT_COUNT.

Regards, Fernando Da Rós

[Former Member]

Hi Fernando,

Thanks for your reply.

I have used the below mentioned statement,

Create FullText Index "MyTable_Index1" On "MyTable"("Concern1")

TEXT ANALYSIS ON

CONFIGURATION 'EXTRACTION_CORE';

Checked the SYS.M_FULLTEXT_QUEUES table, the TOTAL_DOCUMENT_COUNT = 6895;

INDEXED_DOCUMENT_COUNT = 0; QUEUE_DOCUMENT_COUNT = 6891.

Investigated the Preprocessor trace file & looks like this:

BLOB END

[4558]{-1}[-1/-1] 2013-07-23 05:30:03.545715 e preprocessor     PreprocessorPool.cpp(00438) : PreprocessorPool::CreateConnection : Error while creating connection to  server - TrexNet::Exception a protocol mismatch occured, returning to application with errorcode PREPROCESSOR_FAILED

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240292 i Basis            TraceStream.cpp(00383) : ==== Starting hdbpreprocessor, version 1.00.60.379371 (NewDB100_REL), build linuxx86_64 not set 2013-06-27 17:39:32 ld7270.wdf.sap.corp cc (SAP release 20130125, based on SUSE gcc43-4.3.4_20091019-0.22.17.5096.1.PTF.799300) 4.3.4 [gcc-4_3-branch revision 152973]

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240370 i Basis            TraceStream.cpp(00385) : MaxOpenFiles: 1048576

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240385 i Memory           MallocProxy.cpp(01222) : Installed malloc hooks

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240388 i Basis            Timer.cpp(00567) : Using RDTSC for HR timer

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240390 i Memory           AllocatorImpl.cpp(00835) : Allocators activated

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240391 i Memory           AllocatorImpl.cpp(00851) : Using big block segment size 134217728

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240393 i Basis            ProcessorInfo.cpp(00392) : Using GDT segment limit to determine current CPU ID

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240395 w Environment      Environment.cpp(00284) : Changing environment set IMSLERRPATH=/usr/sap/HDB/HDB00/exe//

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240397 w Environment      Environment.cpp(00284) : Changing environment set IMSLSERRPATH=/usr/sap/HDB/HDB00/exe//

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240398 w Environment      Environment.cpp(00284) : Changing environment set NODNSSAPTRANSHOST=1

[4447]{-1}[-1/-1] 2013-07-23 05:33:45.240400 w Environment      Environment.cpp(00263) : Changing environment set SAP_DIR_GLOBAL=/usr/sap/HDB/SYS/global

[6614]{-1}[-1/-1] 2013-07-23 06:11:55.303856 e preprocessor     PreprocessorImpl.cpp(01233) : Text analysis error 'util/factory/src/FactoryImpl.cpp: Line 152: Error 0x0100003E: Unable to register text analysis classes in library /usr/sap/dsod_package//DSoD/bin/libdat-filters-isysconverter41.so.', number of errors: 1

[6614]{-1}[-1/-1] 2013-07-23 06:11:55.303918 e preprocessor     PreprocessorImpl.cpp(00407) : process(): Error in Document, theProcessResult was set to 6618

[6614]{-1}[-1/-1] 2013-07-23 06:11:55.303948 e preprocessor     Preprocessor.cpp(00651) : HANDLE: DISPATCH - Processing Document with key 'MySchema:MyTableen.Concern1:2' failed, returning PREPROCESSOR_ACTIVITY_ERROR (Code 6618)

[6615]{-1}[-1/-1] 2013-07-23 06:11:56.324973 e preprocessor     PreprocessorImpl.cpp(01233) : Text analysis error 'docmodel/analyzer/src/AggregateAnalyzer.cpp: Line 104: Error 0x01000005: Request to create analyzer of unknown type:

SAP.TextAnalysis.DocumentAnalysis.FormatConversion.FormatConversionAnalyzer', number of errors: 1

[6615]{-1}[-1/-1] 2013-07-23 06:11:56.325002 e preprocessor     PreprocessorImpl.cpp(00407) : process(): Error in Document, theProcessResult was set to 6604

[6615]{-1}[-1/-1] 2013-07-23 06:11:56.325020 e preprocessor     Preprocessor.cpp(00651) : HANDLE: DISPATCH - Processing Document with key 'MySchema:MyTableen.Concern1:1' failed, returning PREPROCESSOR_ACTIVITY_ERROR (Code 6604)

[6613]{-1}[-1/-1] 2013-07-23 06:11:57.332431 e preprocessor     PreprocessorImpl.cpp(01233) : Text analysis error 'docmodel/analyzer/src/AggregateAnalyzer.cpp: Line 104: Error 0x01000005: Request to create analyzer of unknown type:

SAP.TextAnalysis.DocumentAnalysis.FormatConversion.FormatConversionAnalyzer', number of errors: 1

[6613]{-1}[-1/-1] 2013-07-23 06:11:57.334338 e preprocessor     PreprocessorImpl.cpp(00407) : process(): Error in Document, theProcessResult was set to 6604

[6613]{-1}[-1/-1] 2013-07-23 06:11:57.334357 e preprocessor     Preprocessor.cpp(00651) : HANDLE: DISPATCH - Processing Document with key 'MySchema:MyTableen.Concern1:3' failed, returning PREPROCESSOR_ACTIVITY_ERROR (Code 6604)

[6615]{-1}[-1/-1] 2013-07-23 06:11:58.339067 e preprocessor     PreprocessorImpl.cpp(01233) : Text analysis error 'docmodel/analyzer/src/AggregateAnalyzer.cpp: Line 104: Error 0x01000005: Request to create analyzer of unknown type:

SAP.TextAnalysis.DocumentAnalysis.FormatConversion.FormatConversionAnalyzer', number of errors: 1

[6615]{-1}[-1/-1] 2013-07-23 06:11:58.339090 e preprocessor     PreprocessorImpl.cpp(00407) : process(): Error in Document, theProcessResult was set to 6604

[6615]{-1}[-1/-1] 2013-07-23 06:11:58.339107 e preprocessor     Preprocessor.cpp(00651) : HANDLE: DISPATCH - Processing Document with key 'MySchema:MyTableen.Concern1:4' failed, returning PREPROCESSOR_ACTIVITY_ERROR (Code 6604)

[6616]{-1}[-1/-1] 2013-07-23 06:11:59.346178 e preprocessor     PreprocessorImpl.cpp(01233) : Text analysis error 'docmodel/analyzer/src/AggregateAnalyzer.cpp: Line 104: Error 0x01000005: Request to create analyzer of unknown type:

SAP.TextAnalysis.DocumentAnalysis.FormatConversion.FormatConversionAnalyzer', number of errors: 1

[6616]{-1}[-1/-1] 2013-07-23 06:11:59.346206 e preprocessor     PreprocessorImpl.cpp(00407) : process(): Error in Document, theProcessResult was set to 6604

[6616]{-1}[-1/-1] 2013-07-23 06:11:59.346223 e preprocessor     Preprocessor.cpp(00651) : HANDLE: DISPATCH - Processing Document with key 'MySchema:MyTableen.Concern1:5' failed, returning PREPROCESSOR_ACTIVITY_ERROR (Code 6604)

[6615]{-1}[-1/-1] 2013-07-23 06:12:00.354276 e preprocessor     PreprocessorImpl.cpp(01233) : Text analysis error 'docmodel/analyzer/src/AggregateAnalyzer.cpp: Line 104: Error 0x01000005: Request to create analyzer of unknown type:

SAP.TextAnalysis.DocumentAnalysis.FormatConversion.FormatConversionAnalyzer', number of errors: 1

In the trace file I see : 

Unable to register text analysis classes in library /usr/sap/dsod_package//DSoD/bin/libdat-filters-isysconverter41.so.

I am not sure how do I set the correct libraries. Any help would be grateful.

Much Thanks,

Indu Nair

[former_member182114]

Hi Indu Nair,

Sorry for delayed response. Looking for your log the most important error I could see is:

"Request to create analyzer of unknown type:

SAP.TextAnalysis.DocumentAnalysis.FormatConversion.FormatConversionAnalyzer"

Please share your statement to create the index and also what you plan to index.

HANA can handle and process PDF, DOC files with some commands but it doesn't seems to be your case.

Regards, Fernando Da Rós

[Former Member]

What was the resolution to this issue?

[Former Member]

Hi Sal,

We were using Hana Rev 60 and it seems the problem that we were getting while creating index is due to come configuration issue in Rev 60.

We have migrated to Rev 56 and it works perfectly fine on it.

Regards,

Indu

[Former Member]

goto the rev 67 first, and try again.

Rev 60 it's it's first rev on SPS6

[Former Member]

All - thanks for the info.  I am using HANA via the AWS Cloud, I am using an image from the SAP Startup forum and they are not on rev67 yet, looks like this is a disadvantage of being on the cloud.  Any workarounds to my current situation on rev60? 

I guess I can spin up the default instance for HANA One offered by AWS - looks like it is rev 52