Calculation discrepancies

Discussions about the Liberty BASIC language, with particular reference to LB Booster
Post Reply
Dermot
Posts: 1
Joined: Thu Apr 19, 2018 1:24 pm

Calculation discrepancies

Post by Dermot » Thu Apr 19, 2018 1:46 pm

I have a POS program written in LBB, and I have the problem where calculations (very rarely) do not appear to be exact.
I show below the sort of thing that happens, and I have used the 'notice' command to display the values

For example, value1 and value2 are the result of calculations involving currency (2 decimals)

the code:
notice str$(value1) + "-" + str$(value2)

results in the following display:

70.52-70.52

as expected, and they appear to be the same, but the code:

if value1 <> value2 then notice str$(value1 - value2)

resuts in the following display:

-6.9388939E-18

ie they look the same but they are not exactly the same, causing the program to take the incorrect course of action

Why does this happen, and how can one prevent it happening?

Dermot

guest
Site Admin
Posts: 56
Joined: Tue Apr 03, 2018 1:34 pm

Re: Calculation discrepancies

Post by guest » Thu Apr 19, 2018 2:05 pm

Dermot wrote:
Thu Apr 19, 2018 1:46 pm
Why does this happen,
I am surprised you should ask, because I thought (by now) this characteristic - of virtually every programming language - was well known. But if you have not encountered the issue before, it happens because numbers are stored internally as floating-point binary, not as decimal. So a number like '70.52', which has a precise value in decimal, does not have a precise value in binary. In fact, should you be interested, the closest LBB can get to that value is:

70.519999999999999996946886682280819513835012912750244140625

That's pretty damn close, but it's not exact. LB 4 has an even greater error, because it works internally with a lower precision than LBB. The closest LB 4 can get to 70.52 is:

70.5199999999999960209606797434389591217041015625
and how can one prevent it happening?
To avoid it happening you should do all your calculations in integers, which are precise. So for example when working with currency values (such as pounds and pence or dollars and cents) you should do all your calculations in integer numbers of pence or cents, and then when you want to display the amount in pounds or dollars you simply divide by 100. Since you're converting to a non-integer value purely for the purposes of display, errors cannot accumulate.

Richard.

bencooper
Posts: 1
Joined: Wed Nov 21, 2018 4:19 pm

Re: Calculation discrepancies

Post by bencooper » Wed Nov 21, 2018 4:37 pm

When searching for an integral solution to a mathematical procedure (for example, a brain teaser), it's not sufficient to use the algorithm
IF (NUM-INT(NUM))= 0 THEN .......,
because if the calculated NUM for an integer NUM includes rounding errors (say 2.000000000001 instead of integer 2) then the comparison fails.

Usually, a slight modification to the above algorithm such as
IF ABS(NUM-INT(NUM))<1E-6 THEN ..... traps the integer while avoiding rounding errors (the ABS function accepts positive
and negative rounding alike).

guest
Site Admin
Posts: 56
Joined: Tue Apr 03, 2018 1:34 pm

Re: Calculation discrepancies

Post by guest » Wed Nov 21, 2018 4:52 pm

bencooper wrote:
Wed Nov 21, 2018 4:37 pm
Usually, a slight modification to the above algorithm...
As you say, "usually". But it's risky, because you are not determining precisely whether a value is an integer but only that it is very close to an integer, and that might not always be good enough. It's far better, when possible, to devise a method that uses only integer calculations and avoids converting to a floating-point value at all; that will maintain 'exact' precision throughout the entire process.

Liberty BASIC and LBB are quite well suited to this kind of calculation, because they both support a wider range of integers than 'typical' BASICs do. LB4 supports 'arbitrary precision' integer arithmetic so you can use values as large as you like with no loss of precision. LBB doesn't, but its integer range (−18,446,744,073,709,551,615 to +18,446,744,073,709,551,615) is still very large.

Post Reply