Andy Robinson wrote:
For the work I do, simply taking the precision of the
most precise input (4.00L)would do the trick, but your
answer (4.0000L) is purer. We should provide a
rounding function, and in practice anyone using such a
function would round (or floor, or ceiling) to get to
the desired precision immediately.
I'm not sure on division either but I'm sure there are
precedents to look at.
The AS400 provides a useful example of the right way to do scaled
decimals.
In the RPG programming language, all internal calculations (i.e.
multiplication, division) are performed to the maximum precision of the
intermediate result (in the multiplication example below), the intermediate
result would be 4.0000L. When the intermediate result is assigned to the
target scaled decimal number, the decimal precision is automatically
extended or truncated to fit the target precision. One extra wrinkle in all
of this is the option to "half-adjust" the intermediate value on assignment;
that is to apply automatic 5/4 rounding to the precision of the target.
So, if the target field is defined as numeric(4,2), the result will
be 4.00L.
These are probably the kind of semantics that a scaled decimal type
would require in Python also; i.e. allow unlimited precision in intermediate
calculations, with a sensible set of rules for assignment to a variable of
different scale and precision.
However, unlike RPG, we should probably ensure that attempts to
overflow or underflow the scale result in NaN or Overflow conditions, rather
than assuming the user is right and losing the significant digits.
Regards,
Mike da Silva
For the work I do, simply taking the precision of the
most precise input (4.00L)would do the trick, but your
answer (4.0000L) is purer. We should provide a
rounding function, and in practice anyone using such a
function would round (or floor, or ceiling) to get to
the desired precision immediately.
I'm not sure on division either but I'm sure there are
precedents to look at.
The AS400 provides a useful example of the right way to do scaled
decimals.
In the RPG programming language, all internal calculations (i.e.
multiplication, division) are performed to the maximum precision of the
intermediate result (in the multiplication example below), the intermediate
result would be 4.0000L. When the intermediate result is assigned to the
target scaled decimal number, the decimal precision is automatically
extended or truncated to fit the target precision. One extra wrinkle in all
of this is the option to "half-adjust" the intermediate value on assignment;
that is to apply automatic 5/4 rounding to the precision of the target.
So, if the target field is defined as numeric(4,2), the result will
be 4.00L.
These are probably the kind of semantics that a scaled decimal type
would require in Python also; i.e. allow unlimited precision in intermediate
calculations, with a sensible set of rules for assignment to a variable of
different scale and precision.
However, unlike RPG, we should probably ensure that attempts to
overflow or underflow the scale result in NaN or Overflow conditions, rather
than assuming the user is right and losing the significant digits.
Regards,
Mike da Silva