document pandas-gbq vision and roadmap

Both pandas-gbq and google-cloud-bigquery are doing many of the same things, and increasingly so (e.g. .to_dataframe() in google-cloud-bigquery)

• Are there different use cases? Can we define those?
• Should we focus development on one and wrap the other? Even if not wholly, for a subset of functionality?
• Is there some direction from Google? @tswast spends a lot of time on both libraries so he is probably best placed to offer guidance

[tswast]

In the interest in not keeping issues open forever, I’m going to treat this issue as a request to document the project vision/roadmap. That should be useful for contributors and also understanding the purpose of this project compared to using the pandas connector in google-cloud-bigquery directly.

[tswast]

To make this task more concrete, I’d like to propose the two following sub-tasks:

• read_gbq calls google-cloud-bigquery’s to_dataframe under the covers. Now that pandas-gbq uses the same logic as pandas for null handling, I don’t expect any change in behavior.
  • I don’t know how we’d implement a progress bar for downloading the dataframe. We may want to upstream the progress bar features (using tqdm) to google-cloud-bigquery library or add some sort of hook so that we can show progress bar.
• to_gbq calls google-cloud-bigquery’s load_table_from_dataframe. load_table_from_dataframe uses Parquet rather CSV but is otherwise quite similar. It may work better with struct and array columns.
  • Logic for overriding the schema will be trickier as the schema is actually defined in the Parquet file.
  • Object columns (and thus nullable types) are not supported by to_parquet in pandas. "Non supported types [for pandas’s to_parquet] include Period and actual Python object types].
  • Perhaps we want to wait on implementing the to_gbq logic until we have a better way to handle nullable columns in load_table_from_dataframe?

With the exception of schema overriding, I think it should be possible to implement these subtasks without changing the public interface of pandas-gbq.