Hi Walter,
Previously when needing e.g. precision of 8, then I would
1. multiply the dividend by 10000,
use the result as a new dividend
2. perform the division using the same divisor but on the new dividend
3. perform my query (If, etc) on the new quotient
The original dividend is untouched.
for example:
21.00 / 5.1 = 4.1176
however,
210000.00/5.1 = 41,176.4706
Hope this helps
Merry
On Tue, 29 Jun 2021 at 01:01, Walter Kiess via Rocket Forum <
Mail@forum.rocketsoftware.com> wrote:
> I've been struggling with this for some time now (years, that is): The
> precision of calculations performed in D3. As everyone knows, D3 performs...
> Be sure to join the forums you're interested in to be notified of new
> content. Click the join button from either the forum listing page or the
> home page of any given sub-forum.
>
> Tip: Want a single update on all your forum memberships? Go to Profile >
> My Account > Forum Notifications, and check 'daily consolidated digest.'
> Switch the discussion email drop down to 'no email' or you will receive
> both.
> ------------------------------
> [image: Rocket Software] <https: community.rocketsoftware.com="">
> D3 and mvBase
> <https: community.rocketsoftware.com/forums/forum-home/digestviewer?tab="digestviewer&CommunityKey=7cdcd1ff-6009-4df2-8b8c-e5ea4293200b">
> Post New Message Online
> <http: community.rocketsoftware.com/participate/postmessage?groupid="55">
> Precision in Numerical Calculations in D3
> <https: community.rocketsoftware.com/forums/forum-home/digestviewer/viewthread?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&CommunityKey=7cdcd1ff-6009-4df2-8b8c-e5ea4293200b&tab=digestviewer#bmf69075be-b108-40a5-b43f-c101ac9551f0">
> Reply to Group Online
> <https: community.rocketsoftware.com/forums/all-discussions/postreply?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&ListKey=365735ee-5669-4857-ac99-8f7ea2cb2c9d"> Reply
> to Group
> <
rocket_d3andmvbase_f69075be-b108-40a5-b43f-c101ac9551f0@connectedcommunity.org?subject=re:+precision+in+numerical+calculations+in+d3>
> [image: Walter Kiess]
> <https: community.rocketsoftware.com/network/members/profile?userkey="4cae0de7-fd8f-4ca0-8f60-f0df6c1c03fd">
> Jun 28, 2021 8:02 PM
> Walter Kiess
> <https: community.rocketsoftware.com/network/members/profile?userkey="4cae0de7-fd8f-4ca0-8f60-f0df6c1c03fd">
> I've been struggling with this for some time now (years, that is): The
> precision of calculations performed in D3.
>
> As everyone knows, D3 performs calculations to a default of 4 decimal
> places. It then truncates the result so everything past the 4th decimal
> place is removed. This in itself is a cause of concern as it introduces the
> 1st of several calculation inaccuracies. From my understanding, this occurs
> for every calculation performed, whether it's in a single expression or in
> separate statements. Thus there may be several inaccuracies in a single
> expression, particularly when fractions or division is involved.
>
> To get around this, I have arranged my expressions such that division is
> always performed last (where possible). But this doesn't solve the problem
> completely though. There is always an intrinsic error in the
> calculation/expression due to the truncation as the expression is not
> calculated as a floating-point expression. When you're dealing with
> hundreds of thousands or even millions of dollars, this becomes a problem,
> as even a small error will have a large effect on the outcome of the
> calculation.
>
> D3 does have the ability to change the precision of its calculations, but
> there is a caveat: ALL programs in the system must use the same PRECISION.
> You can't just call an external subroutine with higher precision to perform
> the calls and return a properly-rounded value. That would cause an error
> due to the different precision of the calling and called programs.
>
> What are developers doing to ensure a higher precision for their
> calculations? What's your "secret sauce"?
>
> ------------------------------
> Walter Kiess
> Manager IT
> SA Police Super
> Adelaide SA Australia
> ------------------------------
> *Reply to Group Online
> <https: community.rocketsoftware.com/forums/all-discussions/postreply?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&ListKey=365735ee-5669-4857-ac99-8f7ea2cb2c9d">*
> *View Thread
> <https: community.rocketsoftware.com/forums/forum-home/digestviewer/viewthread?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&CommunityKey=7cdcd1ff-6009-4df2-8b8c-e5ea4293200b&tab=digestviewer#bmf69075be-b108-40a5-b43f-c101ac9551f0">*
> *Recommend
> <https: community.rocketsoftware.com/forums/forum-home/digestviewer/viewthread?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&CommunityKey=7cdcd1ff-6009-4df2-8b8c-e5ea4293200b&tab=digestviewer&cmd=rate&cmdarg=add#bmf69075be-b108-40a5-b43f-c101ac9551f0">*
> *Forward
> <https: community.rocketsoftware.com/forums/all-discussions/forwardmessages?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&ListKey=365735ee-5669-4857-ac99-8f7ea2cb2c9d">*
> *Flag as Inappropriate
> <https: community.rocketsoftware.com/forums/forum-home/digestviewer/viewthread?messagekey="f69075be-b108-40a5-b43f-c101ac9551f0&CommunityKey=7cdcd1ff-6009-4df2-8b8c-e5ea4293200b&tab=digestviewer&MarkAppropriate=f69075be-b108-40a5-b43f-c101ac9551f0#bmf69075be-b108-40a5-b43f-c101ac9551f0">*
> *Post New Message Online
> <http: community.rocketsoftware.com/participate/postmessage?groupid="55">*
>
>
>
>
> You are subscribed to "D3 and mvBase" as
merry.player@gmail.com. To
> change your subscriptions, go to My Subscriptions
> <https: community.rocketsoftware.com/preferences?section="Subscriptions">.
> To unsubscribe from this community discussion, go to Unsubscribe
> <https: community.rocketsoftware.com/higherlogic/egroups/unsubscribe.aspx?userkey="1096e19a-aadb-4724-8988-3d118b04680e&sKey=KeyRemoved&GroupKey=365735ee-5669-4857-ac99-8f7ea2cb2c9d">
> .
>
--
+44 (0)7774212819
07774212819
https://www.linkedin.com/in/merryplayerhttps://www.merryplayer.com
Original Message:
Sent: 6/28/2021 8:02:00 PM
From: Walter Kiess
Subject: Precision in Numerical Calculations in D3
I've been struggling with this for some time now (years, that is): The precision of calculations performed in D3.
As everyone knows, D3 performs calculations to a default of 4 decimal places. It then truncates the result so everything past the 4th decimal place is removed. This in itself is a cause of concern as it introduces the 1st of several calculation inaccuracies. From my understanding, this occurs for every calculation performed, whether it's in a single expression or in separate statements. Thus there may be several inaccuracies in a single expression, particularly when fractions or division is involved.
To get around this, I have arranged my expressions such that division is always performed last (where possible). But this doesn't solve the problem completely though. There is always an intrinsic error in the calculation/expression due to the truncation as the expression is not calculated as a floating-point expression. When you're dealing with hundreds of thousands or even millions of dollars, this becomes a problem, as even a small error will have a large effect on the outcome of the calculation.
D3 does have the ability to change the precision of its calculations, but there is a caveat: ALL programs in the system must use the same PRECISION. You can't just call an external subroutine with higher precision to perform the calls and return a properly-rounded value. That would cause an error due to the different precision of the calling and called programs.
What are developers doing to ensure a higher precision for their calculations? What's your "secret sauce"?
------------------------------
Walter Kiess
Manager IT
SA Police Super
Adelaide SA Australia
------------------------------
</https:></https:></http:></https:></https:></https:></https:></https:></https:></https:></rocket_d3andmvbase_f69075be-b108-40a5-b43f-c101ac9551f0@connectedcommunity.org?subject=re:+precision+in+numerical+calculations+in+d3></https:></https:></http:></https:></https:>