Read multiple feathers into a single tibble

There are many applications where people break up data into several medium-sized objects. It would be great to be able to read multiple feathers into a single tibble object without an intermediate step.

In the below, the memory used is double what should be necessary:

library(feather)
library(tidyverse)
iris_big <- bind_rows(lapply(1:10000, function(x) iris)) %>% 
  as_tibble()
pryr::object_size(iris_big)
write_feather(iris_big, "tmp1.feather")
write_feather(iris_big, "tmp2.feather")
pryr::mem_change({
  featherdf1 <- read_feather("tmp1.feather")
  featherdf2 <- read_feather("tmp2.feather")
  final_df <- bind_rows(featherdf1, featherdf2)
})

This doesn’t work, but might be how this could go:

pryr::mem_change({
  feather1 <- feather("tmp1.feather")
  feather2 <- feather("tmp2.feather")
  final_df <- bind_rows(feather1, feather2) %>% collect()
})
file.remove("tmp1.feather", "tmp2.feather")

[quinnj]

Note this is currently possible with the Julia Feather.jl package,

julia> using Feather

julia> df = Feather.read(Pkg.dir("Feather") * "/test/newdata/BOD.feather")
6×2 DataFrames.DataFrame
│ Row │ Time │ demand │
├─────┼──────┼────────┤
│ 1   │ 1.0  │ 8.3    │
│ 2   │ 2.0  │ 10.3   │
│ 3   │ 3.0  │ 19.0   │
│ 4   │ 4.0  │ 16.0   │
│ 5   │ 5.0  │ 15.6   │
│ 6   │ 7.0  │ 19.8   │

julia> df = Feather.read(Pkg.dir("Feather") * "/test/newdata/BOD.feather", df; append=true)
12×2 DataFrames.DataFrame
│ Row │ Time │ demand │
├─────┼──────┼────────┤
│ 1   │ 1.0  │ 8.3    │
│ 2   │ 2.0  │ 10.3   │
│ 3   │ 3.0  │ 19.0   │
│ 4   │ 4.0  │ 16.0   │
│ 5   │ 5.0  │ 15.6   │
│ 6   │ 7.0  │ 19.8   │
│ 7   │ 1.0  │ 8.3    │
│ 8   │ 2.0  │ 10.3   │
│ 9   │ 3.0  │ 19.0   │
│ 10  │ 4.0  │ 16.0   │
│ 11  │ 5.0  │ 15.6   │
│ 12  │ 7.0  │ 19.8   │

[wesm]

I created https://issues.apache.org/jira/browse/ARROW-2332 about working on this, let’s continue the discussion there