INT64 uses "C long" type in struct.pack which is 4 bytes.

Expected Behavior

INT64 should be packed as “C long long” to have 64 bits memory capacity.

Current Behavior

INT64 is packed as “C long” which is 32 bits

Steps to reproduce

Materialization of entities with IDs larger than supported range of “C long” raises exception.

Specifications

The bug is in the _serialize_val function in key_encoding_utils.

def _serialize_val(value_type, v: ValueProto) -> Tuple[bytes, int]:
    if value_type == "string_val":
        return v.string_val.encode("utf8"), ValueType.STRING
    elif value_type == "bytes_val":
        return v.bytes_val, ValueType.BYTES
    elif value_type == "int32_val":
        return struct.pack("<i", v.int32_val), ValueType.INT32
    elif value_type == "int64_val":
        return struct.pack("<q", v.int64_val), ValueType.INT64
    else:
        raise ValueError(f"Value type not supported for Firestore: {v}")

• Version:
• feast: 0.18.1
• Platform:
• MacOS 12.0.1
• Subsystem:
• python: 3.8.1

Possible Solution

changing struct.pack("<l", v.int64_val), ValueType.INT64 to struct.pack("<q", v.int64_val), ValueType.INT64 fixes this.

[mjurkus]

We ran into the same issue and we had patched on our fork too in the same way. But that’s not the only issue - the java server doesn’t work with our patch because of this EntityKeySerializerV2.java#L82-L85

Is it really necessary to add backward compatibility for this?

[roelschr]

That makes total sense, thank you for this! I’ll move my PR back to draft and try to work on this backward compatibility issue.