Skip to Content

Query Manager SQL - Decimal Precision

I'm having a real hard time with a formatted search and I figured out that it has to do with the decimal precision. Two values that are multiplied together, if one is a decimal and the other is a number >= 1.0, it works. If both values are < 1.0, it has problems. If I add the two numbers together that are < 1.0, the result is fine. If I multiple two numbers together that are < 1.0 it doesn't. The query below demonstrates this.

declare @OrderTotal float

declare @OrderExp float

declare @DiscPct float

set @OrderTotal = .75

set @DiscPct = 2

set @DiscPct = @DiscPct / 100

set @OrderTotal = @OrderTotal * @DiscPct

select @OrderTotal

Result is 1.5 instead of .15 (.02 x .75)

If one changes the .75 to 1.0, the result is correct a .02

I played around with Float and Real, but I don't know enough about either. Any help would be greatly appreciated.

Add a comment
10|10000 characters needed characters exceeded

Assigned Tags

Related questions

2 Answers

  • author's profile photo Former Member
    Former Member
    Posted on Jun 17, 2008 at 02:26 PM

    .02 x .75 is 0.015 and not 0.15 as you wrote. I run this query and the result is correct, so I think that it belongs to your sql settings.

    Try to change it as

    declare @OrderTotal numeric(9,6)

    declare @OrderExp numeric(9,6)

    declare @DiscPct numeric(9,6)

    set @OrderTotal = .75

    set @DiscPct = 2

    set @DiscPct = @DiscPct / 100

    set @OrderTotal = convert(numeric(9,6), @OrderTotal * @DiscPct)

    select @OrderTotal

    and let me know, if it works

    Add a comment
    10|10000 characters needed characters exceeded

  • Posted on Jun 17, 2008 at 03:39 PM

    Perfect. Thank you very much

    Add a comment
    10|10000 characters needed characters exceeded

Before answering

You should only submit an answer when you are proposing a solution to the poster's problem. If you want the poster to clarify the question or provide more information, please leave a comment instead, requesting additional details. When answering, please include specifics, such as step-by-step instructions, context for the solution, and links to useful resources. Also, please make sure that you answer complies with our Rules of Engagement.
You must be Logged in to submit an answer.

Up to 10 attachments (including images) can be used with a maximum of 1.0 MB each and 10.5 MB total.