Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[morecore] insufficient memory

See original GitHub issue

Hi, The Contigs.txt and NbV1ChF.fasta are 2.8 G and 2.6G. Minimap2 seems to run out of memory on 2TB machine.

$ ragoo.py -t 8 -g 100 -s -b -gff augustus.hints_utr.gff3 Contigs.txt NbV1ChF.fasta
Mon Oct 28 01:45:52 2019 --- Running : minimap2 -k19 -w19 -t8 ../NbV1ChF.fasta ../Contigs.txt > contigs_against_ref.paf 2> contigs_against_ref.paf.log
Mon Oct 28 01:49:23 2019 --- Reading alignments
Mon Oct 28 01:52:45 2019 --- Getting gff features
Mon Oct 28 01:53:07 2019 --- Getting contigs
Mon Oct 28 01:53:25 2019 --- Finding interchromosomally chimeric contigs
Mon Oct 28 01:53:25 2019 --- Finding break points and breaking interchromosomally chimeric contigs
Mon Oct 28 01:53:45 2019 --- Running : minimap2 -k19 -w19 -t8 ../../NbV1ChF.fasta Contigs.inter.chimera.broken.fa > inter_contigs_against_ref.paf 2> inter_contigs_against_ref.paf.log
Mon Oct 28 01:57:22 2019 --- Reading interchromosomal chimera broken alignments
Mon Oct 28 02:00:58 2019 --- Finding intrachromosomally chimeric contigs
Mon Oct 28 02:01:52 2019 --- Running : minimap2 -k19 -w19 -t8 ../../NbV1ChF.fasta Contigs.intra.chimera.broken.fa > intra_contigs_against_ref.paf 2> intra_contigs_against_ref.paf.log
Mon Oct 28 02:05:25 2019 --- Reading intrachromosomal chimera broken alignments
Mon Oct 28 02:09:19 2019 --- The total number of interchromasomally chimeric contigs broken is 0
Mon Oct 28 02:09:19 2019 --- The total number of intrachromasomally chimeric contigs broken is 6
Mon Oct 28 02:09:19 2019 --- Assigning contigs
Mon Oct 28 02:09:40 2019 --- Ordering and orienting contigs
Mon Oct 28 02:11:01 2019 --- Creating pseudomolecules
Mon Oct 28 05:26:02 2019 --- Aligning pseudomolecules to reference
Mon Oct 28 05:26:02 2019 --- Running : minimap2 -ax asm5 --cs -t8 ../../NbV1ChF.fasta ../ragoo.fasta > pm_against_ref.sam 2> pm_contigs_against_ref.sam.log
/bin/bash: line 1: 10389 Aborted                 (core dumped) minimap2 -ax asm5 --cs -t8 ../../NbV1ChF.fasta ../ragoo.fasta > pm_against_ref.sam 2> pm_contigs_against_ref.sam.log
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/ragoo/bin/ragoo.py", line 4, in <module>
    __import__('pkg_resources').run_script('RaGOO==1.1', 'ragoo.py')
  File "/home/ubuntu/miniconda3/envs/ragoo/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/ubuntu/miniconda3/envs/ragoo/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1469, in run_script
    exec(script_code, namespace, namespace)
  File "/home/ubuntu/miniconda3/envs/ragoo/lib/python3.7/site-packages/RaGOO-1.1-py3.7.egg/EGG-INFO/scripts/ragoo.py", line 754, in <module>
  File "/home/ubuntu/miniconda3/envs/ragoo/lib/python3.7/site-packages/RaGOO-1.1-py3.7.egg/EGG-INFO/scripts/ragoo.py", line 439, in align_pms
  File "/home/ubuntu/miniconda3/envs/ragoo/lib/python3.7/site-packages/RaGOO-1.1-py3.7.egg/ragoo_utilities/utilities.py", line 25, in run
RuntimeError: Failed : minimap2 -ax asm5 --cs -t8 ../../NbV1ChF.fasta ../ragoo.fasta > pm_against_ref.sam 2> pm_contigs_against_ref.sam.log

less pm_contigs_against_ref.sam.log 
[M::mm_idx_gen::55.511*1.79] collected minimizers
[M::mm_idx_gen::60.192*2.17] sorted minimizers
[M::main::60.192*2.17] loaded/built the index for 19 target sequence(s)
[M::mm_mapopt_update::64.517*2.10] mid_occ = 342
[M::mm_idx_stat] kmer size: 19; skip: 19; is_hpc: 0; #seq: 19
[M::mm_idx_stat::69.857*2.01] distinct minimizers: 143929053 (83.21% are singletons); average occurrences: 1.928; average spacing: 9.884
[M::worker_pipeline::3024.805*3.40] mapped 5 sequences
[M::worker_pipeline::4921.171*3.81] mapped 6 sequences
[M::worker_pipeline::7761.560*3.75] mapped 4 sequences
[morecore] insufficient memory

How important is the the allignment from Aligning pseudomolecules to reference or can ragoo.fasta been used? How much more memory do you think I would need?

Thank you in advance,

Michal

Issue Analytics

State:
Created 4 years ago
Comments:9 (5 by maintainers)

Top GitHub Comments

1reaction

malongecommented, Apr 20, 2020

Hi there,

I am currently working on v2, which uses pysam to dramatically reduce the memory requirements. In fact, the memory for small and large genomes should be roughly the same.

I am hoping to come out with v2 in the next month or so. I will reopen the issue so that I can send a note when the new version is ready.

Thanks

0reactions

malongecommented, Jun 9, 2020

Hi there,

RagTag, the successor to RaGOO, is now available here:

https://github.com/malonge/RagTag

RagTag now uses pysam to query read coverage, so the memory requirement is dramatically reduced.

Thanks, Mike

Top Results From Across the Web

[morecore] insufficient memory #503 - lh3/minimap2 - GitHub

Hi, I used minimap2 (2.17-r941). The NbV1ChF.fasta and ragoo.fasta are 2.8 G and 3.4 G. However, I ran out of memory on 2...

Moemory error on more cores. - Ansys Learning Forum

Hi,. I am using model with nodes about 5 million with 6 cores( total available 8+ 1 gpu) and 122 gb ram ,....

IV69104: MALLOCTYPE=3.1_64BIT CAN CAUSE CORE ...

With the MALLOCTYPE=3.1_64BIT allocator selected, a core dump can occur during an out-of-memory situation. This has been observed when using dbx with ...

Solutions to error "Insufficient memory to perform operation ...

The message "insufficient memory to perform operation" indicates that your computer does not have enough memory to grid the data with the selected...

Solved in C language Request was made for more information

... if the sbrk() call fails because the system runs out of memory. Do not call sbrk() directly, but instead use the morecore()...