question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DOC: document support for in-memory HDFStore

See original GitHub issue

It seems to be currently imposible to have an in-memory HDFStore.

Any of my attempts at using io.BytesIO failed.

This can be useful for applications where the bytes of the resulting HDF file need to be sent over the network.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
tgycommented, Jun 19, 2020

thanks @dennisbakhuis! let us know if you need help

0reactions
dennisbakhuiscommented, Jun 20, 2020

Do you want the driver parameter to be explicitly mentioned? As it is catched by the **kwargs and passed on to tables.open_file method, it is not really a Pandas parameter.

I currently have added to the docstring that **kwargs passes its parameters to PyTables. Maybe this is too much but I also added an example using **kwargs, passing the driver paramter to create an in-memory HDFStore.

Furthermore, I have added HDFStore class to the reference api, as it was not autogenerated and also made **kwargs more clear that it passes its parameters to PyTables.

Added in the cookbook the method of creating in-memory HDFStores, including an example.

Read more comments on GitHub >

github_iconTop Results From Across the Web

DOC: cookbook for in-memory HDFStores usage · Issue #10491
When writing a DataFrame to hdf format ('to_hdf' method) you cannot pass a file handle instead of a string.
Read more >
kedro.extras.datasets.pandas.hdf_dataset - Read the Docs
Source code for kedro.extras.datasets.pandas.hdf_dataset. """``HDFDataSet`` loads/saves data from/to a hdf file using an underlying ...
Read more >
Pandas, large data, HDF tables and memory usage when ...
When Pandas work on a HDFStore (eg: .mean() or .apply() ), does it load the full data in memory as a DataFrame, or...
Read more >
10.9 HDF5 (PyTables) — Pandas Doc - GitHub Pages
HDFStore supports an top-level API using read_hdf for reading and to_hdf for writing, similar to how read_csv and to_csv work. (new in 0.11.0)....
Read more >
Migrating pandas and local HF5 to HSDS - HDF Forum
Introduction I'm new to the HDF community and just wanted to say that I really appreciate the clear and extensive documentation!
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found