Store HNSW graph connections more compactly

Description

HNSW search is most efficient when all vector data fits in page cache. So good to keep the size of vector files as small as possible.

We currently write all HNSW graph connections as fixed-size integers. This is wasteful since most graphs have far fewer nodes than the max integer value: https://github.com/apache/lucene/blob/d2e22e18c6c92b6a6ba0bbc26d78b5e82832f956/lucene/core/src/java/org/apache/lucene/codecs/lucene94/Lucene94HnswVectorsWriter.java#L478

Maybe instead we could store the connection list using PackedInts.Writer. This would decrease the bits needed to store each connection. We could still ensure that every connection list takes the same number of bytes, to continue being able to index into the graph data easily.

I quickly tested the idea on the 1 million vector GloVe dataset, and saw the graph data size decrease by ~30%:

Baseline
155M	luceneknn-100-16-100.train-16-100.index/_6_Lucene94HnswVectorsFormat_0.vex

Hacky patch
103M	luceneknn-100-16-100.train-16-100.index/_5_Lucene94HnswVectorsFormat_0.vex

[benwtrent]

I changed the PR to move towards delta encoding & vint. Even with storing the memory offsets within vex, the storage improvements are much better than PackedInts.

Table with some numbers around the size improvements for different data sets & parameters:

packed_vex_mb_size	vex_mb_size	packed_index_build_time	index_build_time	params	dataset	percent_reduction
79.9	161.6	767	784	“{‘M’: 16, ‘efConstruction’: 100}”	glove-100-angular	50.55693069
108.4	464.1	1138	1225	“{‘M’: 48, ‘efConstruction’: 100}”	glove-100-angular	76.64296488
2.3	8.2	36	36	“{‘M’: 16, ‘efConstruction’: 100}”	mnist-784-euclidean	71.95121951
2.4	23.5	36	36	“{‘M’: 48, ‘efConstruction’: 100}”	mnist-784-euclidean	89.78723404
66.1	392.2	501	572	“{‘M’: 48, ‘efConstruction’: 100}”	sift-128-euclidean	83.1463539
59.7	136.6	449	516	“{‘M’: 16, ‘efConstruction’: 100}”	sift-128-euclidean	56.29575403

For the curious, here are the QPS numbers (higher is better) for packed (delta & vint) vs baseline:

Glove

MNist

SIFT

[benwtrent]

Could you include a comparison of the search latency and recall numbers you see with this approach? Sometimes with our benchmarks it’s easy to miss small differences in performance.

Yes, I can provide some images and numbers. What I saw when running glove-100-angular batched is that there is no difference. Will run more tests for sure.