Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ParquetFile error not finding _metadata directory

See original GitHub issue

system: centos 7.3, fastparquet 0.1.2, anaconda python 3.6.2

I created some parquet files using

fastparquet.write(filename,data,compression='SNAPPY')

and now when I try to read those same files with

pf=ParquetFile(filename)

I get the error:

NotADirectoryError: [Errno 20] Not a directory: 'filename/_metadata'

thoughts?

Issue Analytics

State:
Created 6 years ago
Comments:16 (7 by maintainers)

Top GitHub Comments

1reaction

Wercurialcommented, Jan 7, 2020

Hello, I have the same problem, but I didn’t understand your solution above. Can you elaborate? Thank you,my fastparquet version is 0.1.5

1reaction

virtuallukecommented, Oct 16, 2017

# ls -al
total 16961
drwxr-xr-x 2 root root     4096 Oct 16 23:34 .
drwxr-xr-x 3 root root     4096 Oct 16 23:34 ..
-rw-r--r-- 1 root root 17359516 Oct 16 23:34 file1.parq

# python
Python 3.6.2 |Anaconda, Inc.| (default, Sep 30 2017, 18:42:57)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastparquet
>>> fastparquet.ParquetFile('file1.parq')
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 96, in __init__
    with open_with(fn2, 'rb') as f:
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/util.py", line 44, in default_open
    return open(f, mode)
NotADirectoryError: [Errno 20] Not a directory: 'file1.parq/_metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 102, in __init__
    self._parse_header(f, verify)
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 125, in _parse_header
    self._set_attrs()
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 139, in _set_attrs
    self.schema = schema.SchemaHelper(self._schema)
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/schema.py", line 80, in __init__
    self.text = schema_to_text(self.schema_elements[0])
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/schema.py", line 47, in schema_to_text
    text += '\n' + schema_to_text(child, indent)
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/schema.py", line 36, in schema_to_text
    root.converted_type])
KeyError: 24