[BUG] Tree Building Failed

Current Behavior

I have run nextstrain build ncov/ with no problems for a few iterations of the GISAID data. Today using the most up to date sequences, the code is failing with the following error:

ERROR: b'/bin/bash: line 1: 18148 Killed                  iqtree -ninit 2 -n 2 -me 0.05 -nt 1 -s results/subsampled_alignment-delim.fasta -m GTR > results/subsampled_alignment-delim.iqtree.log\n'
shell exited 137 when running: iqtree -ninit 2 -n 2 -me 0.05 -nt 1 -s results/subsampled_alignment-delim.fasta -m GTR  > results/subsampled_alignment-delim.iqtree.log

ERROR: TREE BUILDING FAILED
Please see the log file for more details: results/subsampled_alignment-delim.iqtree.log

Building original tree took 537.5226821899414 seconds
[Tue Apr 14 13:20:35 2020]
Error in rule tree:
    jobid: 8
    output: results/tree_raw.nwk
    shell:

        augur tree             --alignment results/subsampled_alignment.fasta             --output results/tree_raw.nwk             --nthreads 1

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /nextstrain/build/.snakemake/log/2020-04-14T124028.174414.snakemake.log

I deleted the repo, re-cloned it and ran nextstrain update to ensure I had the most up to date version of the CLI. Running it all again (including the step to clean the GISAID download with ./scripts/normalize_gisaid_fasta.sh data/gisaid_cov2020_sequences.fasta data/sequences.fasta) I am still getting the same error.

Your environment: if running Nextstrain locally

• Operating system: macOS Catalina
• Version (e.g. auspice 2.7.0): nextstrain.cli 1.16.2

[tsibley]

Thanks for tagging me in @emmahodcroft. 😃

@TrentBrick From the first logs you posted, it looks like Linux’s out-of-memory (OOM) killer in the container is killing the iqtree process when memory is overcommitted:

ERROR: b'/bin/bash: line 1: 18148 Killed iqtree …
shell exited 137 when running: …

When a process is killed by a signal, the shell exits with 128+n where n is the signal number. 137 - 128 = 9, which is SIGKILL (see output of kill -l). This matches the “bash: Killed iqtree” message which is Bash telling you that it noticed iqtree was killed by something.

Can you run nextstrain check-setup and paste the output here? On macOS, Docker runs Linux containers inside a VM. The VM by default only has access to a pretty limited amount of your computer’s memory, and thus the containers are restricted by that same limit. (On Linux, Docker doesn’t have to use a VM, so by default containers have access to all the memory.) nextstrain check-setup will attempt to diagnose the amount of memory available to the Nextstrain container, so its output will be useful to see.

[TrentBrick]

I set Docker to 7Gb and it worked without any issues. Thanks for the rapid and precise help. I don’t know how common this issue is but you may want to flag it in some way for future users?

What was insidious about it was that everything ran well when GISAID had fewer sequences! Either way thanks again.