Heap allocation failure in cslibu_3_0 or crpe32 modules

[dave_smith2]

Using SAP Crystal runtime for .NET v13.0.31.4010

Using .NET Framework 4.8

We have an ASP.NET web application that uses a Windows service. The service has crashed multiple times based on the faulting Windows module ntdll.dll (see below for extract from the Application Log):

Faulting module name: ntdll.dll, version: 10.0.17763.2989, time stamp: 0x21d18902

We opened a support ticket with Microsoft and after collecting various dump files, the engineer found a heap allocation error in one of the two SAP modules cslibu_3_0 or crpe32 that was causing the exception.

I have enclosed excerpts of the dump files and analysis provided by the Microsoft engineer below. Is this an issue that has been detected and resolved in a newer version of the runtime? If not, how can we get this reviewed by the appropriate team at SAP?

Thank you in advance.

Below are excerpts from the call stack traces captured by the dumps:

2. The first is the c0000374 exception which we encountered in earlier logs. Its call stack:

0:036> kc

# Call Site

00 ntdll!ZwWaitForMultipleObjects

01 ntdll!WerpWaitForCrashReporting

…

0e ntdll!RtlpLogHeapFailure

0f ntdll!RtlpSubSegmentAllocate

10 ntdll!RtlpLocalInfoAllocFromCache

11 ntdll!RtlpLowFragHeapAllocFromContext

12 ntdll!RtlpAllocateHeapInternal

13 ucrtbase!_malloc_base

14 mfc140u!operator new

15 cslibu_3_0!CSLib300::CSObArray::setSize

16 cslibu_3_0!CSLib300::CSObArray::add

17 crpe32!MWCleanupProcess

…

3. I can find the same c0000374 exception in other 2 dumps, which has similar (but not the same) call stack:

0:029> kc

# Call Site

00 ntdll!ZwWaitForMultipleObjects

01 ntdll!WerpWaitForCrashReporting

…

0e ntdll!RtlpLogHeapFailure

0f ntdll!RtlpSubSegmentAllocate

10 ntdll!RtlpLocalInfoAllocFromCache

11 ntdll!RtlpLowFragHeapAllocFromContext

12 ntdll!RtlpAllocateHeapInternal

13 ucrtbase!_malloc_base

14 crpe32!MWCleanupProcess

…

And

0:029> kc

# Call Site

00 ntdll!ZwWaitForMultipleObjects

01 ntdll!WerpWaitForCrashReporting

…

0e ntdll!RtlpLogHeapFailure

0f ntdll!RtlpSubSegmentAllocate

10 ntdll!RtlpLocalInfoAllocFromCache

11 ntdll!RtlpLowFragHeapAllocFromContext

12 ntdll!RtlpAllocateHeapInternal

13 ucrtbase!_malloc_base

14 mfc140u!operator new

15 cslibu_3_0!CSLib300::StrDup

16 crpe32!MWCleanupProcess

Further investigation found the source code related to this issue:

if (RtlpIsBlockBusy(HeapEntry) != FALSE) {

RtlpLogHeapFailure(heap_failure_lfh_bitmap_mismatch,

(PVOID)SegmentInfo->LocalData->LowFragHeap->Heap,

HeapEntry,

NULL,

NULL,

NULL);

The cause of this crash is the heap entry allocated shows it is already reserved for unknown reason. We have additional analysis and log file dumps if needed.

From the dumps, we don’t know how and why the UnusedBytes is set to 88.

[former_member11696 · Accepted solution]

As noted, we don't patch this product, full builds only so to find the cause in SP 31 is not an option.

Even if we could replicate the issue it would not be fixed for SP 31.

Your only option is to try SP 32, if it's still an issue I can get DEV to look into it with logging enabled.

Unfortunately single supports cases are no longer available.

You are the only reporting this so it's something with your code causing it possibly.....

All I can suggest is to look into your code and add logging code in your app to see if you can find a common cause.

[former_member11696]

Hi Dave,

Random issues are impossible to track, R&D would need to be able to reproduce the issue on demand.

You are the only one reporting this issue so it's not likely in CR, likely due to your work flow or a Server Issue.

That error suggests something is locked by CR or the OS and it's busy waiting for it to be freed.

I tracked an issue of this sort years ago, CR had marked a temp file to be deleted by the OS, windows marked as deleted so it could use the file name again, the hard drive did not have time to actually delete the file so there was a create file duplicate file name error causing a problem. Not sure what R&D did but it got fixed.

Seem like this is a similar issue.

Check the Windows \temp folder, and delete all files, there should be 5 or so files that are locked but the rest will usually be deleted.

CRPE32, CR's core engine, will create temp files as it's processing reports, once the report is closed the Engine releases all DB connection and instructs the OS to delete the temp files. If the report crashes those temp file, can this "~cre.....tmp" are left behind. Re-booting won't delete the files, you have to manually delete them. Create a batch file to delete them on Windows boot or schedule the batch file when IIS is Cycled.

CR requires Contiguous blocks of memory, so cycle IIS once a day to free up the memory. MS can't defrag memory on the fly like UNIX OS's can.

Searching our source code doesn't return that function, Google searching I found this on MSDN:

https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-rtlcreateheap

So it suggests their could be some issue on your OS, could even be a Memory card issue, CR is hardware and software reliant, so if their is hardware issues it's possible your app may be finding a problem.

In general it does point to a memory issue on your Server.

You could try defragging your Hard Drive, Although I think Windows does this for you but it might help...

Sorry, we used to allow non-contract Users to purchase a single support case but for what ever reason they no longer allow this, only support is through this community now.

Try SP 32, there were other updates that may resolve this...

I'll get R&D to have a read of this post and see if they suggest anything.

Thanks again

Don

[dave_smith2]

Thanks.

We will try these suggestions.

[former_member11696]

FYI

New In SP32 Release - read the above info for SP 21 if upgrading from lower version

1. Microsoft Visual Studio 2022 (64bit)
2. Security updates
3. Addressed customer Incidents
4. New Data source: HANA 2.0 SP06
5. Platform support: Win 10 21H2
6. Platform support: Win 11 21H2 - Check KBA 3204578 for a Windows 11 specific OLE image issue and workaround.
7. Platform support: Chrome Browser version 101

[former_member11696]

Hi Dave,

DEV had a look a the post and suggested first to upgrade to SP 32, because we don't patch this product, full updates only.

If it still happens then they will look into with one of your report that ran into the issue and a test app to view it based on your app code. The reason is because there are things in the report and the API's used can cause specific issues to show up. So emulating your report and code work flow is a must.

They can run the app and see what's happening in the dll that could potentially be the cause.

Do you have a support contract?

If so please log a case and I'll grab it then we can work on this privately?

Let me know what the case number is?

Thanks again

Don

[dave_smith2]

Thanks for the reply.

Our plan now is to go back to an earlier version of the runtime that worked. We are not currently considering moving to SP 32.

We do not have a support contract but would be able to purchase a single support incident if that is offered by SAP.

Is that possible?

Trying to reproduce this behavior on your end using a single report does not seem realistic given the the number of reports that are being used, the complexity of our application and the randomness of the crashes. A better approach might be to provide you with the code flow extracted by the Microsoft engineers indicating where and when the exception took place.

Let me know if that is acceptable.

[former_member11696]

CSLib is own wrapper around memory and other third part dependencies that they won't fix or can't.

So it's hard to say what the source of the issue is.

You need to do more debugging by removing parts of the report to find out the source.

I start by removing any subreports first, then whole sections until it does cause the issue.

[dave_smith2]

I am sorry, but that is not feasible.

We have literally hundreds of reports that are being scheduled. We are not able to pinpoint which report or reports are causing the exception because the crashes occur randomly.

The log dumps clearly show the function in the SAP modules that cause the exception. We can share the findings with your development team. If necessary, we are willing to open a formal support ticket for this purpose.

Further investigation found the source code related to this issue:

if (RtlpIsBlockBusy(HeapEntry) != FALSE) {

RtlpLogHeapFailure(heap_failure_lfh_bitmap_mismatch,

(PVOID)SegmentInfo->LocalData->LowFragHeap->Heap,

HeapEntry,

NULL,

NULL,

NULL);

The cause of this crash is the heap entry allocated shows it is already reserved for unknown reason.

[former_member11696]

OK, try SP 32, as Dell noted there are some bugs in the Viewer in SP 31

[dave_smith2]

We can certainly try SP 32, but has this particular issue been addressed in that service pack?

The reports that cause this exception are being scheduled to various formats like PDF or Excel. They are not being opened in the viewer. How would SP32 resolve those types of reports?

Would it be possible to have your development team look into this based on the dump files Microsoft extracted? There is a lot of good information that indicates where the exception occurs.

[former_member11696]

And what is your app doing when the crash happens?

Do you have 1000's of report objects open?

Are closing all reports once done with them?

[dave_smith2]

No, there are at most a couple of reports being run at any one time.

We close and dispose of all report objects in our application.

I should also mention that we had a similar application using SP24 of the Crystal runtime that behaved without any issues.

[DellSC]

There are known issues in SP31. You should try installing SP32 and see whether that resolves the issue.

-Dell

[dave_smith2]

Thanks for the reply.

Is this a known issue that has been addressed in SP32 or should we upgrade based on a hope and a prayer?

[DellSC]

don.williams, what are your thoughts on this?

-Dell