Filters created with version 1.3.4 and exported to json do not work when imported to version 1.3.9

We use this library in as part of the Ceramic Network. We are publishing batches of work done on the network along with a JSON-serialized bloom filter to make it easier to look up what happened in that batch. Our hosted service making these batches and publishing these bloom filters seems to be using version 1.3.4 of the bloom-filters package. Importing those bloom filters with the current 1.3.9 version of the library causes the .has() method of the reconstituted bloom filter to always return false, even for entries that are definitely present in the filter. Reverting to version 1.3.4 and rebuilding the bloom filter from the same input data causes it to behave correctly. It seems that there was a format change somewhere between version 1.3.4 and version 1.3.9 that causes filters created on the old version, exported with saveAsJSON(), and then imported with fromJSON() on the new version to not build the bloom filter correctly.

Code that builds the filter: https://github.com/ceramicnetwork/ceramic-anchor-service/blob/9ff9e1a20e46c65036fcbed600a0abf1b09d0bec/src/merkle/merkle-objects.ts#L247-L249

[Callidon]

Okay, thanks for your feedback. I’m closing the issue, since we have both fixed it on your sides. Feel free to re-open it if you find any more issues.

[folkvir]

Yes exactly you need to make the versionning yourself. Here the problem is that changes made on how indexes are generated cannot be backward compatible. It does not affect only the BloomFilter but every single class using the old getDistinctIndices function in version <1.3.7. Plus, We have no clue of the version used by people in the exported filter. So the only way for people to continue to use the old system before 1.3.7 is to manually add information about the package version when the filter was created and then must use the _deprecated_getDistinctIndices instead of _getIndexes just after the import.

@Callidon and me disagree to include a versionning system because of this lack of clues and because of the futur code proliferation. But we are ok with a pseudo-backward compatibility of the BloomFilter import because of the optimizations made on the internal storage in the develop branch. Thus, with the old code added in the BaseFilter classe it should be enough for people to use the old index generation while using the new internal structure.