Skip to Content
0
Former Member
Sep 12, 2007 at 08:12 PM

Standard Deviation Issue - more than 65536 records

14 Views

Post Author: SysIntgr

CA Forum: Formula

Is there any known issue calculating the standard deviation in CR9 when you have more than 65536 records. we have more than 100K records and the standard deviation returned by CR is different from the one returned by a SQL query (SQL Server database). to troubleshoot this we started deleting some records at a times and compared the standard deviation values between CR and SQL, they started matching for records 65536 or less, anything more than this number of records returned incorrect value in CR.