Add support for more than 32 shared structures

I created a file that walks through all the possible messages that I have in a system that i’m building. I expected the code below to create the complete structure but it’s only generated for the first 32 message. the structure for messages from tp: 38 are not being generated.

here the exampleMessages from the file: https://www.npoint.io/docs/94e158745cdd8c21f296

import { exampleMessages } from './message/examples';
import * as fs from 'fs';
import { Packr } from 'msgpackr';

const packr = new Packr(
  {
    getStructures() {
      return JSON.parse(fs.readFileSync('./message/structure.json', 'utf8'));
    },
    saveStructures(structures) {
      console.log('create structures', structures);
      fs.writeFileSync('./message/structure.json', JSON.stringify(structures));
    }
  }
);

const main = async  () => {
  for (const exampleMessage of exampleMessages) {
    packr.encode(exampleMessage)
  }

  await new Promise((resolve, _reject) => {
    setTimeout(() => {
      resolve('finish timeout')
    }, 10000)
  })
}

main()

console.log('We created the structure for your messages');

[kriszyp]

Yes, it is by design. The idea is that data may have often have commonly used structures and may also have rarer structures or objects composed of more dynamic keys. And with a limit of shared structures, we can hopefully capture most of the commonly used structures while still being resilient to overloading the shared structures with rare/ad-hoc structures, which may not be reused (or reused infrequently).

This design is also motivated by how structures are encoded. When record structures are enabled and defined, msgpackr defines records with byte encodings that replace the byte encodings for the expansive (I think excessively) range of bytes used for positive integers (0-127). There are 128 some byte encodings available, and I didn’t want to replace the most commonly used positive integers, 0-63, so 64 - 127 are used for records. Of these half (32) are allocated to shared use, and half reserved for use within individual encodings (once you have used up the shared ones, you still want ids available for use within individual data structures).

There are certainly more sophisticated techniques that could be used for differentiating between shared and private structures, but more sophisticating tracking would likely involve more code complexity and could reduce performance. This technique also benefits from being very deterministic.

That being said, probably one of the most straightforward and biggest improvements for situations where there are a lot of data structures that would benefit from being shared, would be to simply allow for two (or more) byte encodings of record ids, which would provide space for vastly more record ids and shared record structures.

Of course, your request to have more flexibility in potentially specifying more shared structures and/or use two byte encodings for more shared structures is quite reasonable, and something I did actually intend to implement at some point. So, I guess I will get to work on adding that 😃.

[kriszyp]

It is a work in progress, definitely not done or even working at all yet.

On Sun, Aug 1, 2021, 8:09 AM Bob Singor @.***> wrote:

@kriszyp https://github.com/kriszyp I see you did this commit. Is this already a working version? Or work in progress?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kriszyp/msgpackr/issues/33#issuecomment-890527588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAIKBSMEJIKWV7DW5L3SLLT2VIQRANCNFSM5BJLHZVQ .