question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Default limit on file input to read_lines is too small

See original GitHub issue

Currently, this is set to default to 128000. This is too small for most practical use, especially given that gs URLs can get quite long. Can the new limit be much higher? We’ll need 5GB (no joke!) for some our larger analyses. Or at least a workflow option to temporarily override?

Otherwise, we get an error such as: "Workflow has invalid declarations: Could not evaluate workflow declarations:\nSingleSampleGenotyping.gvcfs_list:\n\tUse of WdlSingleFile(gs://broad-dsde-methods/gauthier/Finnish_FE_WGS.1000samples.gvcf_list) failed because the file was too big (174730 bytes when only files of up to 128000 bytes are permissible"

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Reactions:2
  • Comments:11 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
danbillscommented, Aug 15, 2018

You want java -Dsystem.input-read-limits.lines=500000 -jar /cromwell-34.jar

For reference you are setting this: https://github.com/broadinstitute/cromwell/blob/0ad9cb61f90276e3576f6ba135cab65c02f09a9e/core/src/main/resources/reference.conf#L157

0reactions
ruchimcommented, Aug 30, 2018

@LeeTL1220 since its possible to configure this limit as needed, I’m hoping you’ve got what you needed. Feel free to reopen if I missed something.

Read more comments on GitHub >

github_iconTop Results From Across the Web

python 3.x - How to make the limit of the readlines to the first ...
If the file is small, just load the whole thing: lines = open('nim.txt').readlines() interesting_lines = lines[1:int(lines[0])+1].
Read more >
<input type="file"> - HTML: HyperText Markup Language | MDN
A file input's value attribute contains a string that represents the ... since doing so will limit the ability of your code to...
Read more >
Python File readlines() Method - W3Schools
The readlines() method returns a list containing each line in the file as a list item. Use the hint parameter to limit the...
Read more >
File uploads with Blazor - Steve Sanderson
By default, SignalR imposes a limit of 32KB for incoming messages, and .NET APIs like Stream.CopyToAsync use much larger internal buffers, so ...
Read more >
How to solve "Too many Open Files" in Java applications
In Linux, the maximum open file limits are set by default for each process or user and the defaut values are quite small....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found