Error on file size check when tbl_df is too large

Hi Hadley and Wes,

Thanks so much for a great collaboration with this project!

When writing and then reading tbl_dfs above a certain size I’m getting the error “Invalid: File is too small to be a well-formed file” (reader.cc line 37). The code below works with 30 million rows, but then fails with 35 million rows (although it did work if I used data.frame). I’ve reproduced it on two different computers and get the same error. I tried installing the development version, but couldn’t get it to work so this is using the CRAN release.

library(feather)
library(dplyr)

test_length <- 35000000
test_df <- data_frame(Area = rep("Area code", test_length),
                      Birthday = rep("1980-01-01", test_length),
                      Age = rep(36, test_length),
                      ID = rep("Identity_code", test_length),
                      Gender = rep("F", test_length),
                      Year = rep(2014, test_length))

write_feather(test_df, "test_df.feather")
read_feather("test_df.feather")

And here’s my session info:

R version 3.3.0 (2016-05-03) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 7 x64 (build 7601) Service Pack 1

locale: [1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] dplyr_0.4.2 feather_0.0.1 devtools_1.11.1

loaded via a namespace (and not attached): [1] Rcpp_0.12.5 digest_0.6.9 withr_1.0.1 assertthat_0.1 R6_2.1.2 DBI_0.3.1
[7] git2r_0.14.0 magrittr_1.5 httr_1.1.0 curl_0.9.7 lazyeval_0.1.10 tools_3.3.0
[13] parallel_3.3.0 memoise_1.0.0 tibble_1.0

[wesm]

(FWIW, the hardest part for me is not having a consistent windows development environment; since Windows is not free even setting up a VM and configuring Visual Studio is a hardship)

[chrisaxelson]

Thank you so much from all the Windows users out there - sorry it was such a mission!