Intermittent Hang in Crystal Reports 10 for Visual Studio 2005

[Former Member]

Our Windows application uses Crystal Reports 10.2 for Visual Studio 2005 (version: 10.2.3600.0).

One of our users is reporting an intermittent hang. It only happens when she runs our application on her machine. If she runs our application on a terminal server, the hang does not occur.

The version of various DLL's on her system:

crpe32.dll - 10.2.0.1210.

CrystalDecisions.CrystalReports.Engine.dll - 10.2.9700.0

I obtained a hang dump and analyzed it with WinDbg. The dump shows 2 interesting threads.

0 - main thread - blocking waiting on a lock

10 - Crystal Reports background thread - waiting for ufmanager.dll to post message to UI thread.

holds 3 locks with these critical sections:

crpe32!SetDialogFont

ufmanager!UFManagerTerminate

<Unloaded_STUI.dll> (contention count 1)

Can someone help me fix this user's hang? I'll be happy to supply the full dump (325 MB uncompressed).

[Former Member · Accepted solution]

Here is some information about the stack dump.

There are 4 managed threads:

0 - main thread - Lock Count 1 - Blocked waiting for lock

2 - finalizer thread

5 - image animator thread

10 - Crystal Reports background thread - Owns lock - Blocked posting back to Thread 0

Using !locks, I see the critical section 0cfab530 is owned by thread 10 (df4):

CritSec <Unloaded_STUI.dll>+cfaa71f at 0cfab530

LockCount 1

RecursionCount 1

OwningThread df4

EntryCount 1

ContentionCount 1

• Locked

Thread 0 Stack Trace (very abbreviated)

00128858 7c90df5a ntdll!ZwWaitForSingleObject+0xc
0012885c 7c91b24b ntdll!RtlpWaitForCriticalSection+0x132, calling ntdll!ZwWaitForSingleObject
00128888 7c80c1d6 kernel32!SetThreadPriority+0x36, calling ntdll!ZwSetInformationThread
0012889c 7c9101db ntdll!RtlAllocateHeap+0xeac, calling ntdll!_SEH_epilog
001288a0 3aa08f8f crpe32!SetDialogFont+0x9f588
001288c0 3a9a5bb9 crpe32!SetDialogFont+0x3c1b2, calling crpe32!SetDialogFont+0x9f50e
001288e4 7c901046 ntdll!RtlEnterCriticalSection+0x46, calling ntdll!RtlpWaitForCriticalSection
001288ec 3a5713d6 crpe32!Ordinal930+0x113d6, calling ntdll!RtlEnterCriticalSection
00128920 3a7dc154 crpe32!PESetBackgroundThreadProc+0x316, calling crpe32!Ordinal930+0x113bc
00128930 3a7da780 crpe32!CRPEConnectionInit+0x1bcb50, calling crpe32!PESetBackgroundThreadProc+0x307
0012894c 3a5b2033 crpe32!PESetSectionHeight+0x6d, calling crpe32!CRPEConnectionInit+0x1bca24
00128964 3a5df5a8 crpe32!PESetLogonProperty+0x47f, calling crpe32!PESetSectionHeight+0x5
...
0012e694 0b40f9b0 (MethodDesc 0xc6e510c +0x60 CrystalDecisions.ReportAppServer.ReportClientDocumentWrapper.Open(System.Object ByRef, Int32)), calling 0ff60666
0012e6b4 0b40f9b0 (MethodDesc 0xc6e510c +0x60 CrystalDecisions.ReportAppServer.ReportClientDocumentWrapper.Open(System.Object ByRef, Int32)), calling 0ff60666
0012e6ec 0b40f7d8 (MethodDesc 0xc6e4eb8 +0x110 CrystalDecisions.ReportAppServer.ReportClientDocumentWrapper.EnsureDocumentIsOpened())
0012e730 0b40ee32 (MethodDesc 0x8effcac +0x362 CrystalDecisions.CrystalReports.Engine.ReportDocument.Load(System.String, CrystalDecisions.Shared.OpenReportMethod, Int16)), calling 011428a0
0012e768 0b40ea25 (MethodDesc 0x8effcd4 +0x55 CrystalDecisions.CrystalReports.Engine.ReportDocument.Load(System.String))
0012e79c 0b40c692 (MethodDesc 0x8efedfc +0x62 profdata.com.Library.libReportRenderCrystal.InitReportDocument())
...

I can see the Thread 0 is blocked on critical section 0cfab530:

0:000> kb100
ChildEBP RetAddr  Args to Child              
00128858 7c90df5a 7c91b24b 00000e24 00000000 ntdll!KiFastSystemCallRet
0012885c 7c91b24b 00000e24 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
001288e4 7c901046 00fab530 3a5713d6 **0cfab530** ntdll!RtlpWaitForCriticalSection+0x132
001288ec 3a5713d6 0cfab530 dbcc4d5a 0cfab528 ntdll!RtlEnterCriticalSection+0x46

Thread 10 Stack Trace (abbreviated)

0ec0e628 7c90df4a ntdll!ZwWaitForMultipleObjects+0xc
0ec0e62c 7c809590 kernel32!WaitForMultipleObjectsEx+0x12c, calling ntdll!NtWaitForMultipleObjects
0ec0e660 7c9100b8 ntdll!RtlpFreeToHeapLookaside+0x22, calling ntdll!RtlpInterlockedPushEntrySList
0ec0e66c 7c910041 ntdll!RtlFreeHeap+0x1e9, calling ntdll!RtlpFreeToHeapLookaside
0ec0e6c8 7e4195f9 user32!RealMsgWaitForMultipleObjectsEx+0x13e, calling kernel32!WaitForMultipleObjectsEx
0ec0e724 7e4196a8 user32!MsgWaitForMultipleObjects+0x1f, calling user32!MsgWaitForMultipleObjectsEx
0ec0e740 0eefdded ufmanager!UFManagerTerminate+0x1677d, calling user32!MsgWaitForMultipleObjects
0ec0e774 7e418b8c user32!NtUserPostMessage+0xc
0ec0e778 0eefd0ca ufmanager!UFManagerTerminate+0x15a5a, calling user32!PostMessageW
0ec0e784 0eefd0dc ufmanager!UFManagerTerminate+0x15a6c, calling ufmanager!UFManagerTerminate+0x16760
0ec0e798 0eefbae4 ufmanager!UFManagerTerminate+0x14474, calling ufmanager!UFManagerTerminate+0x15a00
0ec0e7a0 0eefce55 ufmanager!UFManagerTerminate+0x157e5, calling ufmanager!UFManagerTerminate+0x14460
0ec0e7c8 0eee3c9d ufmanager+0x3c9d
0ec0e860 7c91005d ntdll!RtlFreeHeap+0x647, calling ntdll!_SEH_epilog
0ec0e884 7c9100b8 ntdll!RtlpFreeToHeapLookaside+0x22, calling ntdll!RtlpInterlockedPushEntrySList
...
0ec0feec 3a9b2c29 crpe32!SetDialogFont+0x49222
0ec0ff74 3aa0f912 crpe32!SetDialogFont+0xa5f0b
0ec0ffac 3aa0f9b8 crpe32!SetDialogFont+0xa5fb1, calling crpe32!SetDialogFont+0xa5ef0
0ec0ffb4 7c80b729 kernel32!BaseThreadStart+0x37

Wow, sorry about the loss of formatting.

[Former Member · Accepted solution]

The dump file is 98 MB compressed. I can put it on our FTP site, if you wish to download it.

[former_member11696 · Accepted solution]

Hi Paul,

Forums will not allow attachments. If you need to send it you'll have to purchase a support message and have a dedicated Support Engineer help you debug this.

What you could do is use Process Monitor from www.sysinternals.com and log just your exe. it may show you where the problem is. Talk to their IT guys, maybe its a network issue, across domains or something like that.

Thank you

Don

[Former Member]

SOLUTION

One of our customers was experiencing reproducible hangs running several Crystal Reports in a row. A dump showed a deadlock inside Crystal Reports code.

We tried upgrading to a more recent service pack for Crystal Reports for Visual Studio without any change. We also tried installing Crystal Reports for Visual Studio 2008, and afterwards it seemed she could process twice as many reports before the hang.

Looking at a Process Monitor trace, the last thing our application did was to search through HKEY_CLASSES_ROOT until it hit an entry named "CRUFLParser.Parser". This led me to discover a Crystal Reports feature I hadn't known about-- User Function Libraries (UFL). It turns out the customer had a different application with a registered UFL.

The process monitor trace showed the first successful report would load this UFL OCX, process our report, and then unload the OCX. The next run showed the search through the registry for any UFLs, finding this key, and then nothing.

Uninstalling the other application has solved the problem. Our customer is now able to process reports continually without hanging.

Anyway, I hope this information is of use to you and your developers. I would be very happy if my research could help you avoid a potential deadlock.

Note that I still have 2 hang dumps in case someone wants them. They are about 100 MB zipped.

[former_member11696]

Hi Paul,

Great info. There are 2 flavors of UFL's. They can be COM which requires registering them or using C++ and simply placing the u2l*.dll into our bin folders. CR scan the computer looking for them.

It's odd they were being locked and unloaded but that may be normal depending how your app works. Typically CR keeps them in memory until the Designer is closed.

You probably could have simply un-registered the UFL and your app work work also.

For more info on using and creating UFL's search the Developers help file. We also now fully support UNICODE UFL's in CR 2008 SP1. No need to convert strings to MBCS format.

No need for the dumps. It's likely an issue in the UFL the other program used.

Thanks again

Don