How to force install C version ? Even slower than avro package on Linux CentOS 7.

I installed with pip install fastavro==1.0.0.post1 both on Mac and Linux. Have a simple test to write a same dict 1000 times, compare with avro and json.

start = time.time()
n = 1000
for _ in range(n):
    with ContextStringIO() as f:
        fastavro.schemaless_writer(f, fastavro_schema, row)
end = time.time()
print('fastavro', 1000 * (end - start))

start = time.time()
for _ in range(n):
    writer = avro.io.DatumWriter(schema)
    with ContextStringIO() as f:
        writer.write(row, avro.io.BinaryEncoder(f))
end = time.time()
print('avro', 1000 * (end - start))

start = time.time()
for _ in range(n):
    with ContextStringIO() as f:
        f.write(json.dumps(row).encode())
end = time.time()
print('json', 1000 * (end - start))

Result on Mac:

fastavro 45.388221740722656
avro 327.98099517822266
json 12.218952178955078

fastavro is about 7 times fast than avro, which is very good.

on Linux:

fastavro 479.9647331237793
avro 462.5232219696045
json 13.955354690551758

fastavro is even slower than avro, and 10 times slower than Mac (while avro is only 1.5 times slower).

Tried to see the source code of schemaless_writer from IPython

Mac (use the C version):

In [21]: from fastavro import schemaless_writer

In [22]: schemaless_writer??
Docstring: <no docstring>
Type:      builtin_function_or_method

Linux (use the pure Python version):

In [1]: from fastavro import schemaless_writer

In [2]: schemaless_writer??
Signature: schemaless_writer(fo, schema, record)
Source:
def schemaless_writer(fo, schema, record):
    """Write a single record without the schema or header information

    Parameters
    ----------
    fo: file-like
        Output file
    schema: dict
        Schema
    record: dict
        Record to write


    Example::

        parsed_schema = fastavro.parse_schema(schema)
        with open('file.avro', 'rb') as fp:
            fastavro.schemaless_writer(fp, parsed_schema, record)

    Note: The ``schemaless_writer`` can only write a single record.
    """
    named_schemas = {}
    schema = parse_schema(schema, _named_schemas=named_schemas)

    encoder = BinaryEncoder(fo)
    write_data(encoder, record, schema, named_schemas, "")
    encoder.flush()
File:      ~/.pyenv/versions/weibo/lib/python3.6/site-packages/fastavro/_write_py.py
Type:      function

I also attempted to install via FASTAVRO_USE_CYTHON=1 pip install fastavro, but didn’t work.

[DeanThompson]

Yeah, I reinstalled the Python environment, and it worked. Thanks for helping me to address the problem!

[scottbelden]

I don’t use pyenv so I’m not exactly sure why that file is missing, but looking at https://github.com/pyenv/pyenv/issues/917 and https://github.com/pytorch/pytorch/issues/997 it seems like you might need to pass --enable-shared when you start the environment.