-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Open
Labels
Description
This test code
fprintf(stderr, "%.3g\n", 12.34567);
fprintf(stderr, "%.4g\n", 10.1);currently prints
12.345
10.0999
However, using glibc, this prints
12.3
10.1
We should treat the precision parameter of %g format specifiers as the number of significant digits, not the number of digits after decimal point.
Additionally, it seems like we are incorrectly rounding 10.1 to 10.0999.
See also POSIX:
Let P equal the precision if non-zero, 6 if the precision is omitted, or 1 if the precision is zero. Then, if a conversion with style E would have an exponent of X:
- If P>X>=-4, the conversion shall be with style f (or F) and precision P-(X+1).
- Otherwise, the conversion shall be with style e (or E) and precision P-1.
For 12.345, style E would be printed as 1.235E+01, so the exponent is X=1.
P=3 is greater than X (and >= -4), so it will be printed with style f and precision P-(X+1)=3-(1+1)=1.
Reactions are currently unavailable