Question: does python-rocksdb support importing external SST files, eg for bulk loads?

To optimize for initial large bulk loads, this Rocksdb blog post recommends creating the SST files externally (eg from a big-data pipeline like Spark/MapReduce), and importing them into your DB: http://rocksdb.org/blog/2017/02/17/bulkoad-ingest-sst-file.html

Options options;
SstFileWriter sst_file_writer(EnvOptions(), options, options.comparator);
Status s = sst_file_writer.Open(file_path);
assert(s.ok());

// Insert rows into the SST file, note that inserted keys must be 
// strictly increasing (based on options.comparator)
for (...) {
  s = sst_file_writer.Add(key, value);
  assert(s.ok());
}

// Ingest the external SST file into the DB
s = db_->IngestExternalFile({"/home/usr/file1.sst"}, IngestExternalFileOptions());
assert(s.ok());

The post refers to the C++ Rocksdb API, eg db_->IngestExternalFile().

Does python-rocksdb support this kind of “ingest external SST files” (eg db_->IngestExternalFile()) behavior? I didn’t see this function listed in python-rocksdb. Thanks!

[Congyuwang]

This library has SstFileWriter and ingestExternalFile implemented: https://github.com/Congyuwang/RocksDict.

pip install rocksdict. With pre-build wheels, no need to compile.

Build Write Demo:

from rocksdict import Rdict, Options, SstFileWriter
import random

# generate some rand bytes
rand_bytes1 = [random.randbytes(200) for _ in range(100000)]
rand_bytes1.sort()
rand_bytes2 = [random.randbytes(200) for _ in range(100000)]
rand_bytes2.sort()

# write to file1.sst
writer = SstFileWriter(options=Options(raw_mode=True))
writer.open("file1.sst")
for k, v in zip(rand_bytes1, rand_bytes1):
    writer[k] = v

writer.finish()

# write to file2.sst
writer = SstFileWriter(options=Options(raw_mode=True))
writer.open("file2.sst")
for k, v in zip(rand_bytes2, rand_bytes2):
    writer[k] = v

writer.finish()

# Create a new Rdict with default options
d = Rdict("tmp", options=Options(raw_mode=True))
d.ingest_external_file(["file1.sst", "file2.sst"])
d.close()

# reopen, check if all key-values are there
d = Rdict("tmp", options=Options(raw_mode=True))
for k in rand_bytes2 + rand_bytes1:
    assert d[k] == k

d.close()

# delete tmp
Rdict.destroy("tmp")

[Congyuwang]

Yeah, that’s right. Let me fix it.