MDB_INTEGERKEY usage

Hi,

I’m trying to use ints as keys. As such, I assumed that MDB_INTEGERKEY is the most efficient way to proceed. Unfortunately, I’m getting some keys “out of order”. I say that with quotations because it seems like LMDB is probably doing the correct thing, being the highly used and tested library that it is. This is likely a case of user error, in which case guidance would be much appreciated, or something funky with lmdbjava <-> lmdb. My first guess is that there is something funky going on with the byte ordering.

The int keys 0-255 seem to be in order, but when I transition to 256, it comes before 255.

Here are the relevant details to reproduce: https://gist.github.com/devinrsmith/ccb12bb1cb81dfc88e8f72e60cfb5666

Thanks.

[benalexau]

It might be useful to consider an optional setting so this happens automatically.

The main challenge is LmdbJava’s API tries to be buffer implementation agnostic and the various buffer implementations have distinct methods or idioms to get/set values by byte order. For example ByteBuffer has the order method, byte[] has no inbuilt byte order awareness at all, Agrona’s DirectBuffer offers accessor methods with a byte order argument etc. As such most of the buffers don’t even have a buffer-wide byte order method we could conveniently call.

We could change our BufferProxy.allocate() to BufferProxy.allocate(ByteOrder) and accept null to mean “don’t change it”, but I wonder if the complexity of the corresponding Env.Builder<T>.setDefaultOrder(ByteOrder) would be confusing for users where we cannot set the order anyway. It also seems undesirably asymmetric given users have to set byte orders on the put-time buffers anyway. /cc @krisskross thoughts on this?

[benalexau]

The working solution is below. I added a character to the value to help prove it’s outputting in the correct order. The output is now:

255 000000FF a
256 00000100 b

As shown below with the *** CHANGED *** comments, you just needed to set the key’s byte order to native, and remember to set the byte order on the returned Cursor buffer as well. This is necessary as LmdbJava does not have any opinion on which byte order your buffer used (eg a user who is writing a Java only application storing say string keys and integer values may rely on the Java Big Endian default, and would be surprised if the buffer was set to Little Endian behind the scenes on the returned buffer).

The code:

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.file.Files;
import org.lmdbjava.Cursor;
import org.lmdbjava.Dbi;
import org.lmdbjava.DbiFlags;
import org.lmdbjava.Env;
import org.lmdbjava.Txn;

/**
 * Created by devin on 4/18/17.
 */
public class LMDBIntKey {
    private final ByteBuffer KEY = ByteBuffer.allocateDirect(4);
    private final ByteBuffer VALUE = ByteBuffer.allocateDirect(2);

    private final Env<ByteBuffer> env;
    private final Dbi<ByteBuffer> db;

    public LMDBIntKey() throws IOException {
        final File dir = Files.createTempDirectory(LMDBIntKey.class.getSimpleName()).toFile();
        env = Env.create().open(dir);
        db = env.openDbi(
            LMDBIntKey.class.getSimpleName(),
            DbiFlags.MDB_CREATE,
            DbiFlags.MDB_INTEGERKEY);

        //KEY.order(ByteOrder.BIG_ENDIAN);
        //KEY.order(ByteOrder.LITTLE_ENDIAN);
        KEY.order(ByteOrder.nativeOrder());    /// <---- *** CHANGED ***
    }

    public void execute() {
        KEY.clear();
        KEY.putInt(0x000000FF); // 255
        KEY.flip();
        VALUE.clear();
        VALUE.putChar('a');
        VALUE.flip();
        db.put(KEY, VALUE);

        KEY.clear();
        KEY.putInt(0x00000100); // 256
        KEY.flip();
        VALUE.clear();
        VALUE.putChar('b');
        VALUE.flip();
        db.put(KEY, VALUE);

        try (final Txn<ByteBuffer> txn = env.txnRead()) {
            try (final Cursor<ByteBuffer> cursor = db.openCursor(txn)) {
                if (!cursor.first()) {
                    throw new IllegalStateException("Expected first()");
                }
                cursor.key().order(ByteOrder.nativeOrder());    /// <---- *** CHANGED ***
                System.out.println(String.format("%d %08X %c",
                    cursor.key().getInt(0),
                    cursor.key().getInt(0),
                    cursor.val().getChar(0)));
                if (!cursor.next()) {
                    throw new IllegalStateException("Expected next()");
                }
                System.out.println(String.format("%d %08X %c",
                    cursor.key().getInt(0),
                    cursor.key().getInt(0),
                    cursor.val().getChar(0)));
            }
        }
    }

    public static void main(String[] args) throws IOException {
        final LMDBIntKey wal = new LMDBIntKey();
        wal.execute();
    }
}

Thanks for trying LmdbJava. I hope you find it useful.