We have a website that creates PDF-reports using SAP Crystal Reports runtime engine for .NET Framework 4 (64-bit) SP1. The site runs on Windows 2003 R2 x64.
The site (w3wp.exe) crashes every now and then when a report is generated in PDF. We cannot trace it back to specific reports, users or actions. It seems to be random. We don't have this issue on x86-webservers with the same application and CR-version.
We got a memory dump from one of the crashes. The dump can be found [here|http://www.megaupload.com/?d=ZFVJ7FEZ] (zipped, about 330MB). Microsoft already analyzed this dump and this is the conclusion :
Here is my analysis: as you already noted, the stack goes from the SAP component to msvcr80, based on the method name (without the private symbols we cannot see the arguments passed etcu2026) it looks like they are initializing something which requires memory to be moved around, hence the call to memmove_s
Child-SP RetAddr Call Site
00000000`27ceec88 00000000`781b6a70 msvcr80!memcpy(unsigned char * dst = 0x00000000`26bbceb8 "???", unsigned char * src = 0x00000000`26bbceb8 "???", unsigned long count = 0)+0x1ec
00000000`27ceec90 00000000`41d69db4 msvcr80!memmove_s(void * dst = 0x00000000`00000000, unsigned int64 sizeInBytes = 2, void * src = 0x00000000`000001c2, unsigned int64 count = 0)+0x80
00000000`27ceecd0 00000000`41d6b2bc crxf_pdf!UXFInitializeW+0x1834
00000000`27ceed50 00000000`37b2d78f crxf_pdf!UXFGetExportFormatsExW+0x10c
00000000`27ceeda0 00000000`37da9c1a crpe32!CRPEConnectionInit+0x365df
00000000`27ceeef0 00000000`37da0b93 crpe32!MWCleanupProcess+0x4ffca
00000000`27ceefd0 00000000`37da54eb crpe32!MWCleanupProcess+0x46f43
00000000`27cef720 00000000`37d27bb4 crpe32!MWCleanupProcess+0x4b89b
00000000`27cef750 00000000`37d38964 crpe32!CRPEConnectionInit+0x230a04
00000000`27cef820 00000000`37d39a8f crpe32!CRPEConnectionInit+0x2417b4
00000000`27cefb90 00000000`37d39dac crpe32!CRPEConnectionInit+0x2428df
00000000`27cefbe0 00000000`37d3a45c crpe32!CRPEConnectionInit+0x242bfc
00000000`27cefd30 00000000`783917cd crpe32!CRPEConnectionInit+0x2432ac
00000000`27cefe20 00000000`781337d7 mfc80u!_AfxThreadEntry(void * pParam = 0x00000000`23e01f80)+0x101
00000000`27ceff20 00000000`78133894 msvcr80!_callthreadstartex(void)+0x17
00000000`27ceff50 00000000`77d6b71a msvcr80!_threadstartex(void * ptd = 0x00000000`77d6b6e0)+0x84
00000000`27ceff80 00000000`00000000 kernel32!BaseThreadStart(<function> * lpStartAddress = 0x00000000`00000000, void * lpParameter = 0x00000000`00000000)+0x3a
First, itu2019s interesting to note that the destination pointer does not look to be valid; the call to msvcr80!memmove_s comes directly from crxf_pdf!UXFInitializeW (there are no u201Chiddenu201D calls in the middle):
0:076> ln 00000000`41d69db4
(00000000`41d68580) crxf_pdf!UXFInitializeW+0x1834 | (00000000`41d6b1b0) crxf_pdf!UXFGetExportFormatsExW
The exception (an Access Violation) is actually thrown in msvcr80 while it is trying to copy the memory as a result of the move call:
0:076> .exr -1
ExceptionAddress: 000000007814e38c (msvcr80!memcpy+0x00000000000001ec)
ExceptionCode: c0000005 (Access violation)
Attempt to write to address 000000002c78b280
Also looking at the thread registers confirms that the memory address is not valid:
rax=0000000000000009 rbx=0000000000000413 rcx=000000002c78b280
rdx=ffffffffff9329f0 rsi=0000000000000000 rdi=0000000000000002
rip=000000007814e38c rsp=0000000027ceec88 rbp=0000000041dd42b0
r8=0000000005500dc0 r9=0000000005500dc4 r10=000000002728a4c0
r11=000000002728a4c0 r12=0000000041dd42b0 r13=0000000000000001
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010204
00000000`7814e38c 8901 mov dword ptr [rcx],eax ds:00000000`2c78b280=????????
Unfortunately without the private symbols we cannot see what that memory belongs to.
Anyway since I have access to all Microsoft private symbols, the fact that I cannot find this specific one I think means this is part of the SAP components. I think we can conclude the problem originates from the crxf_pdf.dll and manifests itself in msvcr80.dll because of some wrong parameter passed into that call.
All help would be very much appreciated.