Discussion: Float encoding

At the risk of causing bikeshedding, I wanted to raise a point of discussion regarding how floats are encoded – given that integers are encoded in Base62 and therefore take up less space than their decimal form, it seems a bit odd that floats are left as-is.

My first idea was that they could also be encoded in a similar compacted form, such as base36, which JavaScript can do natively with .toString(36), but that isn’t particularly great if the float’s magnitude is not close to 0 because the expontent e+n notation can’t be used unambiguously, and parsing them back from such a format probably isn’t easy (definitely not in JavaScript.)

I’d love to hear more about the rationale behind this decision (if any)! All in all, for the goals that it sets out to accomplish, JCOF is really neat.

(Edit, this did end up causing bikeshedding, especially in the other issues, sorry for that.)

[mortie]

A proper float to string algorithm doesn’t just choose some number of sufficient digits; the string "2.2" represents exactly the one floating point number which is the closest to being 2.2, and a float parser will return that exact floating point value when parsing "2.2". And the expression 1/3 returns a floating point value; (1/3).toString() returns the shortest string representation where (1/3) is the closest float, so that the float parser returns the exact same float.

My problem is that I only know about these challenges, not how they’re solved, or even how difficult they are to solve.

FWIW, I tried out the algorithms in https://observablehq.com/d/e3010313d892b36e, and a lot of numbers aren’t round-tripped correctly. For example, the base 10 float 1.1344e-319 stringifies to "f2WY-5d", which parses to 1.12094e-319, and 9.399999999999983 stringifies to "fH3egP4fQc-f" which parses to 9.399999999999984.

Here’s a snippet which can be used to test all the floats:

// Return the next representable double from value towards direction
// From https://stackoverflow.com/a/27661477
function nextNearest(value) {
  if (!isFinite(value))
    return value;
  
  var buffer = new ArrayBuffer(8);
  var f64 = new Float64Array(buffer);
  var u32 = new Uint32Array(buffer);
  
  f64[0] = value;
  
  if (value === 0) {
    u32[0] = 1;
    u32[1] = 0;
  } else if ((value > 0) && (value < Infinity) || (value < 0) && (value > Infinity)) {
    if (u32[0]++ === 0xFFFFFFFF)
      u32[1]++;
  } else {
    if (u32[0]-- === 0)
      u32[1]--;
  }
  
  return f64[0];
}

let f = 0;
while (f != Infinity) {
  let str = stringifyF62(f);
  let parsed = parseF62(str);
  if (parsed == f) {
    console.log("OK:", f, "from string", str);
  } else {
    console.log("ERR: expected", f, "got", parsed, "from string", str);
  }

  f = nextNearest(f);
}

Replace with stringifyF62(f) with f.toString() and parseF62(str) with parseFloat(str), and you can see that the JavaScript float stringify and parse functions round-trip all numbers you try perfectly.

[JobLeonard]

… actually, given how I currently coded it we could replace i, I, f and F prefixes with a universal n and N prefix, since the optional ., ,, + and - separators can be used to indicate if we’re dealing with a pure integer or with a floating point number.