Default limit on file input to read_lines is too small
See original GitHub issueCurrently, this is set to default to 128000. This is too small for most practical use, especially given that gs URLs can get quite long. Can the new limit be much higher? We’ll need 5GB (no joke!) for some our larger analyses. Or at least a workflow option to temporarily override?
Otherwise, we get an error such as:
"Workflow has invalid declarations: Could not evaluate workflow declarations:\nSingleSampleGenotyping.gvcfs_list:\n\tUse of WdlSingleFile(gs://broad-dsde-methods/gauthier/Finnish_FE_WGS.1000samples.gvcf_list) failed because the file was too big (174730 bytes when only files of up to 128000 bytes are permissible"
Issue Analytics
- State:
- Created 6 years ago
- Reactions:2
- Comments:11 (7 by maintainers)
Top Results From Across the Web
python 3.x - How to make the limit of the readlines to the first ...
If the file is small, just load the whole thing: lines = open('nim.txt').readlines() interesting_lines = lines[1:int(lines[0])+1].
Read more ><input type="file"> - HTML: HyperText Markup Language | MDN
A file input's value attribute contains a string that represents the ... since doing so will limit the ability of your code to...
Read more >Python File readlines() Method - W3Schools
The readlines() method returns a list containing each line in the file as a list item. Use the hint parameter to limit the...
Read more >File uploads with Blazor - Steve Sanderson
By default, SignalR imposes a limit of 32KB for incoming messages, and .NET APIs like Stream.CopyToAsync use much larger internal buffers, so ...
Read more >How to solve "Too many Open Files" in Java applications
In Linux, the maximum open file limits are set by default for each process or user and the defaut values are quite small....
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
You want
java -Dsystem.input-read-limits.lines=500000 -jar /cromwell-34.jar
For reference you are setting this: https://github.com/broadinstitute/cromwell/blob/0ad9cb61f90276e3576f6ba135cab65c02f09a9e/core/src/main/resources/reference.conf#L157
@LeeTL1220 since its possible to configure this limit as needed, I’m hoping you’ve got what you needed. Feel free to reopen if I missed something.