Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Snappy Issue while using SortSam
See original GitHub issueBug Report
Affected tool(s)
PicardTools SortSam
Affected version(s)
- [x ] Latest public release version [2.18.14]
- 2.17.4 is also affected
- 1.134
Description
Errors happens 2 out of 3 times. Here are example logs:
Log1
[boursin@node10 ~]$ /cm/shared/bioinfo/java/1.8.0_92/bin/java -Xmx18000m -Dsnappy.loader.verbosity=true -Djava.io.tmpdir=/data/tmp -Dorg.xerial.snappy.lib.path=/usr/lib64/libsnappy.so.1.1.4 -jar /cm/shared/bioinfo//picard/2.17.4/picard.jar SortSam CREATE_INDEX=false INPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.2.bam OUTPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.2.sorted.bam SORT_ORDER=coordinate TMP_DIR=/data/tmp VERBOSITY=DEBUG
21:53:04.741 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/cm/shared/bioinfo/picard/2.17.4/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Sat Oct 13 21:53:04 CEST 2018] SortSam INPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.2.bam OUTPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.2.sorted.bam SORT_ORDER=coordinate TMP_DIR=[/data/tmp] VERBOSITY=DEBUG CREATE_INDEX=false QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Sat Oct 13 21:53:04 CEST 2018] Executing as boursin@node10 on Linux 3.10.0-229.4.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14; Deflater: Intel; Inflater: Intel; Picard version: 2.17.4-SNAPSHOT
DEBUG 2018-10-13 21:53:04 BlockCompressedOutputStream Using deflater: IntelDeflater
DEBUG 2018-10-13 21:53:07 SnappyLoader Snappy successfully loaded.
INFO 2018-10-13 21:53:56 SortSam Read 10,000,000 records. Elapsed time: 00:00:51s. Time for last 10,000,000: 51s. Last read position: chr15:88,863,504
INFO 2018-10-13 21:54:50 SortSam Read 20,000,000 records. Elapsed time: 00:01:45s. Time for last 10,000,000: 54s. Last read position: chr8:104,617,748
INFO 2018-10-13 21:55:43 SortSam Read 30,000,000 records. Elapsed time: 00:02:38s. Time for last 10,000,000: 52s. Last read position: chr9:121,965,993
INFO 2018-10-13 21:56:35 SortSam Read 40,000,000 records. Elapsed time: 00:03:30s. Time for last 10,000,000: 52s. Last read position: chrM:3,603
INFO 2018-10-13 21:57:28 SortSam Read 50,000,000 records. Elapsed time: 00:04:23s. Time for last 10,000,000: 52s. Last read position: chr4:156,047,399
INFO 2018-10-13 21:58:21 SortSam Read 60,000,000 records. Elapsed time: 00:05:17s. Time for last 10,000,000: 53s. Last read position: chr8:120,160,321
INFO 2018-10-13 21:59:11 SortSam Read 70,000,000 records. Elapsed time: 00:06:06s. Time for last 10,000,000: 49s. Last read position: chr10:116,113,279
INFO 2018-10-13 22:00:05 SortSam Read 80,000,000 records. Elapsed time: 00:07:00s. Time for last 10,000,000: 54s. Last read position: chr17:34,650,291
INFO 2018-10-13 22:00:58 SortSam Read 90,000,000 records. Elapsed time: 00:07:53s. Time for last 10,000,000: 52s. Last read position: chr4:154,152,546
INFO 2018-10-13 22:01:50 SortSam Read 100,000,000 records. Elapsed time: 00:08:45s. Time for last 10,000,000: 52s. Last read position: chr4:131,780,062
INFO 2018-10-13 22:02:42 SortSam Read 110,000,000 records. Elapsed time: 00:09:37s. Time for last 10,000,000: 51s. Last read position: chr14:4,350,723
INFO 2018-10-13 22:03:34 SortSam Read 120,000,000 records. Elapsed time: 00:10:29s. Time for last 10,000,000: 51s. Last read position: chr7:98,492,472
INFO 2018-10-13 22:04:31 SortSam Read 130,000,000 records. Elapsed time: 00:11:26s. Time for last 10,000,000: 56s. Last read position: chr14:50,737,024
INFO 2018-10-13 22:05:25 SortSam Read 140,000,000 records. Elapsed time: 00:12:20s. Time for last 10,000,000: 54s. Last read position: chr4:42,850,249
INFO 2018-10-13 22:06:22 SortSam Read 150,000,000 records. Elapsed time: 00:13:17s. Time for last 10,000,000: 57s. Last read position: chr9:46,277,499
INFO 2018-10-13 22:07:12 SortSam Read 160,000,000 records. Elapsed time: 00:14:07s. Time for last 10,000,000: 49s. Last read position: chr7:58,813,631
INFO 2018-10-13 22:08:09 SortSam Read 170,000,000 records. Elapsed time: 00:15:04s. Time for last 10,000,000: 57s. Last read position: chr2:181,092,150
INFO 2018-10-13 22:09:04 SortSam Read 180,000,000 records. Elapsed time: 00:15:59s. Time for last 10,000,000: 54s. Last read position: chrX:20,637,486
INFO 2018-10-13 22:09:58 SortSam Read 190,000,000 records. Elapsed time: 00:16:53s. Time for last 10,000,000: 54s. Last read position: chr2:11,409,170
INFO 2018-10-13 22:10:49 SortSam Read 200,000,000 records. Elapsed time: 00:17:45s. Time for last 10,000,000: 51s. Last read position: chr4:135,465,627
INFO 2018-10-13 22:11:44 SortSam Read 210,000,000 records. Elapsed time: 00:18:39s. Time for last 10,000,000: 54s. Last read position: chr5:136,222,008
INFO 2018-10-13 22:12:39 SortSam Read 220,000,000 records. Elapsed time: 00:19:34s. Time for last 10,000,000: 55s. Last read position: chr9:29,411,744
INFO 2018-10-13 22:13:31 SortSam Read 230,000,000 records. Elapsed time: 00:20:27s. Time for last 10,000,000: 52s. Last read position: chr15:89,557,558
INFO 2018-10-13 22:14:32 SortSam Read 240,000,000 records. Elapsed time: 00:21:27s. Time for last 10,000,000: 60s. Last read position: chr2:155,709,064
INFO 2018-10-13 22:15:28 SortSam Read 250,000,000 records. Elapsed time: 00:22:24s. Time for last 10,000,000: 56s. Last read position: chr9:20,941,657
INFO 2018-10-13 22:16:22 SortSam Read 260,000,000 records. Elapsed time: 00:23:17s. Time for last 10,000,000: 53s. Last read position: chr13:88,821,859
INFO 2018-10-13 22:17:13 SortSam Read 270,000,000 records. Elapsed time: 00:24:08s. Time for last 10,000,000: 51s. Last read position: chr1:93,105,210
INFO 2018-10-13 22:18:09 SortSam Read 280,000,000 records. Elapsed time: 00:25:04s. Time for last 10,000,000: 55s. Last read position: chr10:76,581,546
INFO 2018-10-13 22:19:01 SortSam Read 290,000,000 records. Elapsed time: 00:25:56s. Time for last 10,000,000: 52s. Last read position: chr7:141,699,617
INFO 2018-10-13 22:19:52 SortSam Read 300,000,000 records. Elapsed time: 00:26:47s. Time for last 10,000,000: 50s. Last read position: chr9:83,688,474
INFO 2018-10-13 22:20:47 SortSam Read 310,000,000 records. Elapsed time: 00:27:42s. Time for last 10,000,000: 55s. Last read position: chr13:38,955,920
INFO 2018-10-13 22:21:40 SortSam Read 320,000,000 records. Elapsed time: 00:28:35s. Time for last 10,000,000: 53s. Last read position: chr2:150,586,385
INFO 2018-10-13 22:22:32 SortSam Read 330,000,000 records. Elapsed time: 00:29:27s. Time for last 10,000,000: 51s. Last read position: chr9:108,362,837
INFO 2018-10-13 22:23:24 SortSam Read 340,000,000 records. Elapsed time: 00:30:19s. Time for last 10,000,000: 52s. Last read position: chr2:98,666,818
INFO 2018-10-13 22:24:16 SortSam Read 350,000,000 records. Elapsed time: 00:31:11s. Time for last 10,000,000: 52s. Last read position: chr12:115,862,030
INFO 2018-10-13 22:25:09 SortSam Read 360,000,000 records. Elapsed time: 00:32:04s. Time for last 10,000,000: 52s. Last read position: chr19:5,570,366
INFO 2018-10-13 22:25:59 SortSam Read 370,000,000 records. Elapsed time: 00:32:54s. Time for last 10,000,000: 49s. Last read position: chr15:28,247,498
INFO 2018-10-13 22:26:03 SortSam Finished reading inputs, merging and writing to output now.
INFO 2018-10-13 22:29:20 SortSam Wrote 10,000,000 records from a sorting collection. Elapsed time: 00:36:15s. Time for last 10,000,000: 154s. Last read position: chr1:87,389,279
INFO 2018-10-13 22:31:37 SortSam Wrote 20,000,000 records from a sorting collection. Elapsed time: 00:38:32s. Time for last 10,000,000: 137s. Last read position: chr1:180,935,153
INFO 2018-10-13 22:34:11 SortSam Wrote 30,000,000 records from a sorting collection. Elapsed time: 00:41:06s. Time for last 10,000,000: 153s. Last read position: chr2:60,422,678
INFO 2018-10-13 22:37:01 SortSam Wrote 40,000,000 records from a sorting collection. Elapsed time: 00:43:56s. Time for last 10,000,000: 170s. Last read position: chr2:145,934,434
INFO 2018-10-13 22:39:35 SortSam Wrote 50,000,000 records from a sorting collection. Elapsed time: 00:46:30s. Time for last 10,000,000: 153s. Last read position: chr3:36,671,235
INFO 2018-10-13 22:42:19 SortSam Wrote 60,000,000 records from a sorting collection. Elapsed time: 00:49:14s. Time for last 10,000,000: 164s. Last read position: chr3:126,842,587
INFO 2018-10-13 22:44:56 SortSam Wrote 70,000,000 records from a sorting collection. Elapsed time: 00:51:51s. Time for last 10,000,000: 157s. Last read position: chr4:101,526,887
INFO 2018-10-13 22:47:26 SortSam Wrote 80,000,000 records from a sorting collection. Elapsed time: 00:54:21s. Time for last 10,000,000: 150s. Last read position: chr4:140,758,199
INFO 2018-10-13 22:49:53 SortSam Wrote 90,000,000 records from a sorting collection. Elapsed time: 00:56:48s. Time for last 10,000,000: 146s. Last read position: chr5:30,934,373
INFO 2018-10-13 22:52:43 SortSam Wrote 100,000,000 records from a sorting collection. Elapsed time: 00:59:38s. Time for last 10,000,000: 170s. Last read position: chr5:116,131,362
INFO 2018-10-13 22:55:14 SortSam Wrote 110,000,000 records from a sorting collection. Elapsed time: 01:02:09s. Time for last 10,000,000: 150s. Last read position: chr5:146,301,587
INFO 2018-10-13 22:58:02 SortSam Wrote 120,000,000 records from a sorting collection. Elapsed time: 01:04:57s. Time for last 10,000,000: 168s. Last read position: chr6:91,787,965
INFO 2018-10-13 23:00:51 SortSam Wrote 130,000,000 records from a sorting collection. Elapsed time: 01:07:46s. Time for last 10,000,000: 169s. Last read position: chr7:7,958,228
INFO 2018-10-13 23:03:27 SortSam Wrote 140,000,000 records from a sorting collection. Elapsed time: 01:10:22s. Time for last 10,000,000: 155s. Last read position: chr7:30,313,770
INFO 2018-10-13 23:05:58 SortSam Wrote 150,000,000 records from a sorting collection. Elapsed time: 01:12:53s. Time for last 10,000,000: 151s. Last read position: chr7:73,409,547
INFO 2018-10-13 23:08:29 SortSam Wrote 160,000,000 records from a sorting collection. Elapsed time: 01:15:24s. Time for last 10,000,000: 150s. Last read position: chr7:127,893,272
INFO 2018-10-13 23:11:50 SortSam Wrote 170,000,000 records from a sorting collection. Elapsed time: 01:18:45s. Time for last 10,000,000: 201s. Last read position: chr8:47,261,717
INFO 2018-10-13 23:14:42 SortSam Wrote 180,000,000 records from a sorting collection. Elapsed time: 01:21:37s. Time for last 10,000,000: 171s. Last read position: chr8:110,223,053
INFO 2018-10-13 23:17:22 SortSam Wrote 190,000,000 records from a sorting collection. Elapsed time: 01:24:17s. Time for last 10,000,000: 160s. Last read position: chr9:45,952,847
INFO 2018-10-13 23:20:18 SortSam Wrote 200,000,000 records from a sorting collection. Elapsed time: 01:27:13s. Time for last 10,000,000: 176s. Last read position: chr9:110,088,056
INFO 2018-10-13 23:23:32 SortSam Wrote 210,000,000 records from a sorting collection. Elapsed time: 01:30:27s. Time for last 10,000,000: 193s. Last read position: chr10:77,063,940
INFO 2018-10-13 23:26:22 SortSam Wrote 220,000,000 records from a sorting collection. Elapsed time: 01:33:17s. Time for last 10,000,000: 169s. Last read position: chr11:3,129,453
INFO 2018-10-13 23:29:21 SortSam Wrote 230,000,000 records from a sorting collection. Elapsed time: 01:36:16s. Time for last 10,000,000: 178s. Last read position: chr11:61,069,802
INFO 2018-10-13 23:32:27 SortSam Wrote 240,000,000 records from a sorting collection. Elapsed time: 01:39:22s. Time for last 10,000,000: 185s. Last read position: chr11:97,551,875
INFO 2018-10-13 23:35:30 SortSam Wrote 250,000,000 records from a sorting collection. Elapsed time: 01:42:25s. Time for last 10,000,000: 183s. Last read position: chr11:120,654,275
[Sat Oct 13 23:37:39 CEST 2018] picard.sam.SortSam done. Elapsed time: 104.58 minutes.
Runtime.totalMemory()=1840775168
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: Read error; BinaryCodec in readmode; streamed file (filename not available)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:420)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:394)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:380)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:209)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:37)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.advance(SortingCollection.java:544)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.next(SortingCollection.java:534)
at htsjdk.samtools.util.PeekIterator.peek(PeekIterator.java:69)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:570)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:565)
at java.util.TreeMap.put(TreeMap.java:552)
at java.util.TreeSet.add(TreeSet.java:255)
at htsjdk.samtools.util.SortingCollection$MergingIterator.next(SortingCollection.java:478)
at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:225)
at picard.sam.SortSam.doWork(SortSam.java:167)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:269)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)
Caused by: java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:98)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:634)
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:167)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:418)
... 17 more
Log 2
[boursin@node10 ~]$ singularity run -B /data -C docker://broadinstitute/picard SortSam CREATE_INDEX=false INPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam OUTPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam SORT_ORDER=coordinate TMP_DIR=/data/tmp VERBOSITY=DEBUG
Docker image path: index.docker.io/broadinstitute/picard:latest
Cache folder set to /home/boursin/.singularity/docker
Creating container runtime...
Exploding layer: sha256:e5ad7970bc69d2ba62d2bbb4b80627273350e1982ff43bc1ca04a3215bc6da5b.tar.gz
Exploding layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Exploding layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Exploding layer: sha256:027ff82bff61d36ab0c8ea43c0b609f5685e59e3ed12928d331273e17a842b8e.tar.gz
Exploding layer: sha256:43d0dc9fe721272287ff81a42a233f67d86d19ea25456ba8190a5dfa7c0cba6d.tar.gz
Exploding layer: sha256:ff898a018f90edc552f5adb92551e31c20d44e170f4cda7585b48fe303f05f39.tar.gz
Exploding layer: sha256:c947a016ff64812602b309cbb5be1b6d54ba0f29ea52ad75a7d363d263361ea0.tar.gz
Exploding layer: sha256:28acff2b65f1a9a0d99caf9bcf557fc4672064dabb23008dba257c83852c3e7c.tar.gz
Exploding layer: sha256:6aead6da9ea09903e35c843189e727b3881ab0b64f05103d7055ce6819a9a4df.tar.gz
Exploding layer: sha256:27223ae9526089abab5c4d46a010add6b447887e8cdfe1857026b6651b4b6cb6.tar.gz
INFO 2018-10-13 17:23:26 SortSam
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** SortSam -CREATE_INDEX false -INPUT /data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam -OUTPUT /data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam -SORT_ORDER coordinate -TMP_DIR /data/tmp -VERBOSITY DEBUG
**********
17:23:26.567 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/usr/picard/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Sat Oct 13 17:23:26 UTC 2018] SortSam INPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam OUTPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam SORT_ORDER=coordinate TMP_DIR=[/data/tmp] VERBOSITY=DEBUG CREATE_INDEX=false QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Sat Oct 13 17:23:26 UTC 2018] Executing as boursin@node10 on Linux 3.10.0-229.4.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: unspecified-SNAPSHOT
DEBUG 2018-10-13 17:23:27 BlockCompressedOutputStream Using deflater: IntelDeflater
DEBUG 2018-10-13 17:23:30 SnappyLoader Snappy successfully loaded.
INFO 2018-10-13 17:24:15 SortSam Read 10,000,000 records. Elapsed time: 00:00:48s. Time for last 10,000,000: 48s. Last read position: chr15:88,863,504
INFO 2018-10-13 17:25:01 SortSam Read 20,000,000 records. Elapsed time: 00:01:34s. Time for last 10,000,000: 45s. Last read position: chr8:104,617,748
INFO 2018-10-13 17:25:46 SortSam Read 30,000,000 records. Elapsed time: 00:02:18s. Time for last 10,000,000: 44s. Last read position: chr9:121,965,993
INFO 2018-10-13 17:26:32 SortSam Read 40,000,000 records. Elapsed time: 00:03:04s. Time for last 10,000,000: 46s. Last read position: chrM:3,603
INFO 2018-10-13 17:27:17 SortSam Read 50,000,000 records. Elapsed time: 00:03:49s. Time for last 10,000,000: 45s. Last read position: chr4:156,047,399
INFO 2018-10-13 17:28:01 SortSam Read 60,000,000 records. Elapsed time: 00:04:33s. Time for last 10,000,000: 43s. Last read position: chr8:120,160,321
INFO 2018-10-13 17:28:49 SortSam Read 70,000,000 records. Elapsed time: 00:05:21s. Time for last 10,000,000: 47s. Last read position: chr10:116,113,279
INFO 2018-10-13 17:29:34 SortSam Read 80,000,000 records. Elapsed time: 00:06:06s. Time for last 10,000,000: 45s. Last read position: chr17:34,650,291
INFO 2018-10-13 17:30:19 SortSam Read 90,000,000 records. Elapsed time: 00:06:52s. Time for last 10,000,000: 45s. Last read position: chr4:154,152,546
INFO 2018-10-13 17:31:03 SortSam Read 100,000,000 records. Elapsed time: 00:07:35s. Time for last 10,000,000: 43s. Last read position: chr4:131,780,062
INFO 2018-10-13 17:31:48 SortSam Read 110,000,000 records. Elapsed time: 00:08:21s. Time for last 10,000,000: 45s. Last read position: chr14:4,350,723
INFO 2018-10-13 17:32:33 SortSam Read 120,000,000 records. Elapsed time: 00:09:05s. Time for last 10,000,000: 44s. Last read position: chr7:98,492,472
INFO 2018-10-13 17:33:22 SortSam Read 130,000,000 records. Elapsed time: 00:09:54s. Time for last 10,000,000: 49s. Last read position: chr14:50,737,024
INFO 2018-10-13 17:34:09 SortSam Read 140,000,000 records. Elapsed time: 00:10:41s. Time for last 10,000,000: 46s. Last read position: chr4:42,850,249
INFO 2018-10-13 17:34:53 SortSam Read 150,000,000 records. Elapsed time: 00:11:25s. Time for last 10,000,000: 44s. Last read position: chr9:46,277,499
INFO 2018-10-13 17:35:39 SortSam Read 160,000,000 records. Elapsed time: 00:12:12s. Time for last 10,000,000: 46s. Last read position: chr7:58,813,631
INFO 2018-10-13 17:36:28 SortSam Read 170,000,000 records. Elapsed time: 00:13:00s. Time for last 10,000,000: 48s. Last read position: chr2:181,092,150
INFO 2018-10-13 17:37:13 SortSam Read 180,000,000 records. Elapsed time: 00:13:45s. Time for last 10,000,000: 45s. Last read position: chrX:20,637,486
INFO 2018-10-13 17:38:03 SortSam Read 190,000,000 records. Elapsed time: 00:14:35s. Time for last 10,000,000: 49s. Last read position: chr2:11,409,170
INFO 2018-10-13 17:38:51 SortSam Read 200,000,000 records. Elapsed time: 00:15:23s. Time for last 10,000,000: 48s. Last read position: chr4:135,465,627
INFO 2018-10-13 17:39:38 SortSam Read 210,000,000 records. Elapsed time: 00:16:10s. Time for last 10,000,000: 47s. Last read position: chr5:136,222,008
INFO 2018-10-13 17:40:27 SortSam Read 220,000,000 records. Elapsed time: 00:16:59s. Time for last 10,000,000: 48s. Last read position: chr9:29,411,744
INFO 2018-10-13 17:41:11 SortSam Read 230,000,000 records. Elapsed time: 00:17:43s. Time for last 10,000,000: 44s. Last read position: chr15:89,557,558
INFO 2018-10-13 17:41:58 SortSam Read 240,000,000 records. Elapsed time: 00:18:30s. Time for last 10,000,000: 46s. Last read position: chr2:155,709,064
INFO 2018-10-13 17:42:44 SortSam Read 250,000,000 records. Elapsed time: 00:19:16s. Time for last 10,000,000: 46s. Last read position: chr9:20,941,657
INFO 2018-10-13 17:43:27 SortSam Read 260,000,000 records. Elapsed time: 00:19:59s. Time for last 10,000,000: 43s. Last read position: chr13:88,821,859
INFO 2018-10-13 17:44:13 SortSam Read 270,000,000 records. Elapsed time: 00:20:45s. Time for last 10,000,000: 45s. Last read position: chr1:93,105,210
INFO 2018-10-13 17:44:56 SortSam Read 280,000,000 records. Elapsed time: 00:21:28s. Time for last 10,000,000: 43s. Last read position: chr10:76,581,546
INFO 2018-10-13 17:45:43 SortSam Read 290,000,000 records. Elapsed time: 00:22:15s. Time for last 10,000,000: 46s. Last read position: chr7:141,699,617
INFO 2018-10-13 17:46:31 SortSam Read 300,000,000 records. Elapsed time: 00:23:03s. Time for last 10,000,000: 47s. Last read position: chr9:83,688,474
INFO 2018-10-13 17:47:19 SortSam Read 310,000,000 records. Elapsed time: 00:23:51s. Time for last 10,000,000: 48s. Last read position: chr13:38,955,920
INFO 2018-10-13 17:48:05 SortSam Read 320,000,000 records. Elapsed time: 00:24:38s. Time for last 10,000,000: 46s. Last read position: chr2:150,586,385
INFO 2018-10-13 17:48:54 SortSam Read 330,000,000 records. Elapsed time: 00:25:26s. Time for last 10,000,000: 48s. Last read position: chr9:108,362,837
INFO 2018-10-13 17:49:38 SortSam Read 340,000,000 records. Elapsed time: 00:26:11s. Time for last 10,000,000: 44s. Last read position: chr2:98,666,818
INFO 2018-10-13 17:50:28 SortSam Read 350,000,000 records. Elapsed time: 00:27:00s. Time for last 10,000,000: 49s. Last read position: chr12:115,862,030
INFO 2018-10-13 17:51:16 SortSam Read 360,000,000 records. Elapsed time: 00:27:48s. Time for last 10,000,000: 48s. Last read position: chr19:5,570,366
INFO 2018-10-13 17:51:59 SortSam Read 370,000,000 records. Elapsed time: 00:28:31s. Time for last 10,000,000: 42s. Last read position: chr15:28,247,498
INFO 2018-10-13 17:52:03 SortSam Finished reading inputs, merging and writing to output now.
INFO 2018-10-13 17:52:04 SortingCollection Creating merging iterator from 742 files
INFO 2018-10-13 17:55:42 SortSam Wrote 10,000,000 records from a sorting collection. Elapsed time: 00:32:14s. Time for last 10,000,000: 165s. Last read position: chr1:87,389,279
INFO 2018-10-13 17:58:26 SortSam Wrote 20,000,000 records from a sorting collection. Elapsed time: 00:34:58s. Time for last 10,000,000: 164s. Last read position: chr1:180,935,153
INFO 2018-10-13 18:01:08 SortSam Wrote 30,000,000 records from a sorting collection. Elapsed time: 00:37:40s. Time for last 10,000,000: 162s. Last read position: chr2:60,422,678
INFO 2018-10-13 18:04:25 SortSam Wrote 40,000,000 records from a sorting collection. Elapsed time: 00:40:57s. Time for last 10,000,000: 197s. Last read position: chr2:145,934,434
INFO 2018-10-13 18:07:07 SortSam Wrote 50,000,000 records from a sorting collection. Elapsed time: 00:43:39s. Time for last 10,000,000: 161s. Last read position: chr3:36,671,235
INFO 2018-10-13 18:10:10 SortSam Wrote 60,000,000 records from a sorting collection. Elapsed time: 00:46:43s. Time for last 10,000,000: 183s. Last read position: chr3:126,842,587
INFO 2018-10-13 18:13:25 SortSam Wrote 70,000,000 records from a sorting collection. Elapsed time: 00:49:57s. Time for last 10,000,000: 194s. Last read position: chr4:101,526,887
INFO 2018-10-13 18:16:06 SortSam Wrote 80,000,000 records from a sorting collection. Elapsed time: 00:52:38s. Time for last 10,000,000: 161s. Last read position: chr4:140,758,199
INFO 2018-10-13 18:19:02 SortSam Wrote 90,000,000 records from a sorting collection. Elapsed time: 00:55:34s. Time for last 10,000,000: 176s. Last read position: chr5:30,934,373
INFO 2018-10-13 18:22:47 SortSam Wrote 100,000,000 records from a sorting collection. Elapsed time: 00:59:19s. Time for last 10,000,000: 224s. Last read position: chr5:116,131,362
INFO 2018-10-13 18:25:36 SortSam Wrote 110,000,000 records from a sorting collection. Elapsed time: 01:02:08s. Time for last 10,000,000: 168s. Last read position: chr5:146,301,587
INFO 2018-10-13 18:28:22 SortSam Wrote 120,000,000 records from a sorting collection. Elapsed time: 01:04:54s. Time for last 10,000,000: 165s. Last read position: chr6:91,787,965
INFO 2018-10-13 18:31:33 SortSam Wrote 130,000,000 records from a sorting collection. Elapsed time: 01:08:05s. Time for last 10,000,000: 190s. Last read position: chr7:7,958,228
INFO 2018-10-13 18:34:51 SortSam Wrote 140,000,000 records from a sorting collection. Elapsed time: 01:11:23s. Time for last 10,000,000: 198s. Last read position: chr7:30,313,770
INFO 2018-10-13 18:38:03 SortSam Wrote 150,000,000 records from a sorting collection. Elapsed time: 01:14:35s. Time for last 10,000,000: 191s. Last read position: chr7:73,409,547
INFO 2018-10-13 18:41:02 SortSam Wrote 160,000,000 records from a sorting collection. Elapsed time: 01:17:35s. Time for last 10,000,000: 179s. Last read position: chr7:127,893,272
INFO 2018-10-13 18:44:31 SortSam Wrote 170,000,000 records from a sorting collection. Elapsed time: 01:21:03s. Time for last 10,000,000: 208s. Last read position: chr8:47,261,717
INFO 2018-10-13 18:47:34 SortSam Wrote 180,000,000 records from a sorting collection. Elapsed time: 01:24:06s. Time for last 10,000,000: 182s. Last read position: chr8:110,223,053
INFO 2018-10-13 18:50:40 SortSam Wrote 190,000,000 records from a sorting collection. Elapsed time: 01:27:12s. Time for last 10,000,000: 186s. Last read position: chr9:45,952,847
INFO 2018-10-13 18:53:59 SortSam Wrote 200,000,000 records from a sorting collection. Elapsed time: 01:30:31s. Time for last 10,000,000: 198s. Last read position: chr9:110,088,056
INFO 2018-10-13 18:57:12 SortSam Wrote 210,000,000 records from a sorting collection. Elapsed time: 01:33:45s. Time for last 10,000,000: 193s. Last read position: chr10:77,063,940
INFO 2018-10-13 19:00:22 SortSam Wrote 220,000,000 records from a sorting collection. Elapsed time: 01:36:54s. Time for last 10,000,000: 189s. Last read position: chr11:3,129,453
INFO 2018-10-13 19:03:18 SortSam Wrote 230,000,000 records from a sorting collection. Elapsed time: 01:39:51s. Time for last 10,000,000: 176s. Last read position: chr11:61,069,802
[Sat Oct 13 19:03:45 UTC 2018] picard.sam.SortSam done. Elapsed time: 100.31 minutes.
Runtime.totalMemory()=6603931648
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: Read error; BinaryCodec in readmode; streamed file (filename not available)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:420)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:394)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:380)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:268)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:40)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.advance(SortingCollection.java:626)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.next(SortingCollection.java:616)
at htsjdk.samtools.util.PeekIterator.peek(PeekIterator.java:69)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:653)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:648)
at java.util.TreeMap.put(TreeMap.java:552)
at java.util.TreeSet.add(TreeSet.java:255)
at htsjdk.samtools.util.SortingCollection$MergingIterator.next(SortingCollection.java:562)
at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:223)
at picard.sam.SortSam.doWork(SortSam.java:167)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:295)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)
Caused by: java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:98)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:634)
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:167)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:418)
... 17 more
Log 3
[boursin@node10 ~]$ singularity run -B /data -C docker://broadinstitute/picard SortSam CREATE_INDEX=false INPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam OUTPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam SORT_ORDER=coordinate TMP_DIR=/data/tmp
Docker image path: index.docker.io/broadinstitute/picard:latest
Cache folder set to /home/boursin/.singularity/docker
Creating container runtime...
Exploding layer: sha256:e5ad7970bc69d2ba62d2bbb4b80627273350e1982ff43bc1ca04a3215bc6da5b.tar.gz
Exploding layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Exploding layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Exploding layer: sha256:027ff82bff61d36ab0c8ea43c0b609f5685e59e3ed12928d331273e17a842b8e.tar.gz
Exploding layer: sha256:43d0dc9fe721272287ff81a42a233f67d86d19ea25456ba8190a5dfa7c0cba6d.tar.gz
Exploding layer: sha256:ff898a018f90edc552f5adb92551e31c20d44e170f4cda7585b48fe303f05f39.tar.gz
Exploding layer: sha256:c947a016ff64812602b309cbb5be1b6d54ba0f29ea52ad75a7d363d263361ea0.tar.gz
Exploding layer: sha256:28acff2b65f1a9a0d99caf9bcf557fc4672064dabb23008dba257c83852c3e7c.tar.gz
Exploding layer: sha256:6aead6da9ea09903e35c843189e727b3881ab0b64f05103d7055ce6819a9a4df.tar.gz
Exploding layer: sha256:27223ae9526089abab5c4d46a010add6b447887e8cdfe1857026b6651b4b6cb6.tar.gz
INFO 2018-10-13 15:13:22 SortSam
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** SortSam -CREATE_INDEX false -INPUT /data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam -OUTPUT /data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam -SORT_ORDER coordinate -TMP_DIR /data/tmp
**********
15:13:23.205 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/usr/picard/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Sat Oct 13 15:13:23 UTC 2018] SortSam INPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam OUTPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam SORT_ORDER=coordinate TMP_DIR=[/data/tmp] CREATE_INDEX=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Sat Oct 13 15:13:23 UTC 2018] Executing as boursin@node10 on Linux 3.10.0-229.4.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: unspecified-SNAPSHOT
INFO 2018-10-13 15:14:12 SortSam Read 10,000,000 records. Elapsed time: 00:00:47s. Time for last 10,000,000: 47s. Last read position: chr15:88,863,504
INFO 2018-10-13 15:14:59 SortSam Read 20,000,000 records. Elapsed time: 00:01:34s. Time for last 10,000,000: 46s. Last read position: chr8:104,617,748
INFO 2018-10-13 15:15:44 SortSam Read 30,000,000 records. Elapsed time: 00:02:19s. Time for last 10,000,000: 44s. Last read position: chr9:121,965,993
INFO 2018-10-13 15:16:28 SortSam Read 40,000,000 records. Elapsed time: 00:03:03s. Time for last 10,000,000: 44s. Last read position: chrM:3,603
INFO 2018-10-13 15:17:11 SortSam Read 50,000,000 records. Elapsed time: 00:03:46s. Time for last 10,000,000: 42s. Last read position: chr4:156,047,399
INFO 2018-10-13 15:17:56 SortSam Read 60,000,000 records. Elapsed time: 00:04:31s. Time for last 10,000,000: 44s. Last read position: chr8:120,160,321
INFO 2018-10-13 15:18:41 SortSam Read 70,000,000 records. Elapsed time: 00:05:16s. Time for last 10,000,000: 45s. Last read position: chr10:116,113,279
INFO 2018-10-13 15:19:25 SortSam Read 80,000,000 records. Elapsed time: 00:06:00s. Time for last 10,000,000: 43s. Last read position: chr17:34,650,291
INFO 2018-10-13 15:20:11 SortSam Read 90,000,000 records. Elapsed time: 00:06:46s. Time for last 10,000,000: 46s. Last read position: chr4:154,152,546
INFO 2018-10-13 15:20:55 SortSam Read 100,000,000 records. Elapsed time: 00:07:30s. Time for last 10,000,000: 43s. Last read position: chr4:131,780,062
INFO 2018-10-13 15:21:41 SortSam Read 110,000,000 records. Elapsed time: 00:08:15s. Time for last 10,000,000: 45s. Last read position: chr14:4,350,723
INFO 2018-10-13 15:22:27 SortSam Read 120,000,000 records. Elapsed time: 00:09:02s. Time for last 10,000,000: 46s. Last read position: chr7:98,492,472
INFO 2018-10-13 15:23:13 SortSam Read 130,000,000 records. Elapsed time: 00:09:48s. Time for last 10,000,000: 45s. Last read position: chr14:50,737,024
INFO 2018-10-13 15:23:56 SortSam Read 140,000,000 records. Elapsed time: 00:10:31s. Time for last 10,000,000: 43s. Last read position: chr4:42,850,249
INFO 2018-10-13 15:24:41 SortSam Read 150,000,000 records. Elapsed time: 00:11:16s. Time for last 10,000,000: 45s. Last read position: chr9:46,277,499
INFO 2018-10-13 15:25:28 SortSam Read 160,000,000 records. Elapsed time: 00:12:03s. Time for last 10,000,000: 46s. Last read position: chr7:58,813,631
INFO 2018-10-13 15:26:17 SortSam Read 170,000,000 records. Elapsed time: 00:12:52s. Time for last 10,000,000: 49s. Last read position: chr2:181,092,150
INFO 2018-10-13 15:27:06 SortSam Read 180,000,000 records. Elapsed time: 00:13:41s. Time for last 10,000,000: 48s. Last read position: chrX:20,637,486
INFO 2018-10-13 15:27:52 SortSam Read 190,000,000 records. Elapsed time: 00:14:27s. Time for last 10,000,000: 46s. Last read position: chr2:11,409,170
INFO 2018-10-13 15:28:40 SortSam Read 200,000,000 records. Elapsed time: 00:15:15s. Time for last 10,000,000: 48s. Last read position: chr4:135,465,627
INFO 2018-10-13 15:29:26 SortSam Read 210,000,000 records. Elapsed time: 00:16:01s. Time for last 10,000,000: 45s. Last read position: chr5:136,222,008
INFO 2018-10-13 15:30:13 SortSam Read 220,000,000 records. Elapsed time: 00:16:47s. Time for last 10,000,000: 46s. Last read position: chr9:29,411,744
INFO 2018-10-13 15:30:56 SortSam Read 230,000,000 records. Elapsed time: 00:17:31s. Time for last 10,000,000: 43s. Last read position: chr15:89,557,558
INFO 2018-10-13 15:31:45 SortSam Read 240,000,000 records. Elapsed time: 00:18:20s. Time for last 10,000,000: 49s. Last read position: chr2:155,709,064
INFO 2018-10-13 15:32:28 SortSam Read 250,000,000 records. Elapsed time: 00:19:03s. Time for last 10,000,000: 42s. Last read position: chr9:20,941,657
INFO 2018-10-13 15:33:12 SortSam Read 260,000,000 records. Elapsed time: 00:19:47s. Time for last 10,000,000: 44s. Last read position: chr13:88,821,859
INFO 2018-10-13 15:33:57 SortSam Read 270,000,000 records. Elapsed time: 00:20:32s. Time for last 10,000,000: 44s. Last read position: chr1:93,105,210
INFO 2018-10-13 15:34:42 SortSam Read 280,000,000 records. Elapsed time: 00:21:17s. Time for last 10,000,000: 44s. Last read position: chr10:76,581,546
INFO 2018-10-13 15:35:27 SortSam Read 290,000,000 records. Elapsed time: 00:22:02s. Time for last 10,000,000: 45s. Last read position: chr7:141,699,617
INFO 2018-10-13 15:36:14 SortSam Read 300,000,000 records. Elapsed time: 00:22:48s. Time for last 10,000,000: 46s. Last read position: chr9:83,688,474
INFO 2018-10-13 15:36:59 SortSam Read 310,000,000 records. Elapsed time: 00:23:34s. Time for last 10,000,000: 45s. Last read position: chr13:38,955,920
INFO 2018-10-13 15:37:44 SortSam Read 320,000,000 records. Elapsed time: 00:24:19s. Time for last 10,000,000: 44s. Last read position: chr2:150,586,385
INFO 2018-10-13 15:38:28 SortSam Read 330,000,000 records. Elapsed time: 00:25:03s. Time for last 10,000,000: 44s. Last read position: chr9:108,362,837
INFO 2018-10-13 15:39:12 SortSam Read 340,000,000 records. Elapsed time: 00:25:47s. Time for last 10,000,000: 44s. Last read position: chr2:98,666,818
INFO 2018-10-13 15:39:58 SortSam Read 350,000,000 records. Elapsed time: 00:26:33s. Time for last 10,000,000: 45s. Last read position: chr12:115,862,030
INFO 2018-10-13 15:40:45 SortSam Read 360,000,000 records. Elapsed time: 00:27:20s. Time for last 10,000,000: 46s. Last read position: chr19:5,570,366
INFO 2018-10-13 15:41:30 SortSam Read 370,000,000 records. Elapsed time: 00:28:05s. Time for last 10,000,000: 45s. Last read position: chr15:28,247,498
INFO 2018-10-13 15:41:35 SortSam Finished reading inputs, merging and writing to output now.
INFO 2018-10-13 15:41:35 SortingCollection Creating merging iterator from 742 files
INFO 2018-10-13 15:44:39 SortSam Wrote 10,000,000 records from a sorting collection. Elapsed time: 00:31:13s. Time for last 10,000,000: 141s. Last read position: chr1:87,389,279
INFO 2018-10-13 15:46:52 SortSam Wrote 20,000,000 records from a sorting collection. Elapsed time: 00:33:27s. Time for last 10,000,000: 133s. Last read position: chr1:180,935,153
INFO 2018-10-13 15:49:18 SortSam Wrote 30,000,000 records from a sorting collection. Elapsed time: 00:35:53s. Time for last 10,000,000: 146s. Last read position: chr2:60,422,678
INFO 2018-10-13 15:52:09 SortSam Wrote 40,000,000 records from a sorting collection. Elapsed time: 00:38:44s. Time for last 10,000,000: 170s. Last read position: chr2:145,934,434
INFO 2018-10-13 15:54:35 SortSam Wrote 50,000,000 records from a sorting collection. Elapsed time: 00:41:10s. Time for last 10,000,000: 146s. Last read position: chr3:36,671,235
INFO 2018-10-13 15:57:07 SortSam Wrote 60,000,000 records from a sorting collection. Elapsed time: 00:43:42s. Time for last 10,000,000: 152s. Last read position: chr3:126,842,587
INFO 2018-10-13 16:00:03 SortSam Wrote 70,000,000 records from a sorting collection. Elapsed time: 00:46:38s. Time for last 10,000,000: 175s. Last read position: chr4:101,526,887
INFO 2018-10-13 16:02:26 SortSam Wrote 80,000,000 records from a sorting collection. Elapsed time: 00:49:01s. Time for last 10,000,000: 142s. Last read position: chr4:140,758,199
INFO 2018-10-13 16:04:29 SortSam Wrote 90,000,000 records from a sorting collection. Elapsed time: 00:51:04s. Time for last 10,000,000: 122s. Last read position: chr5:30,934,373
INFO 2018-10-13 16:07:21 SortSam Wrote 100,000,000 records from a sorting collection. Elapsed time: 00:53:55s. Time for last 10,000,000: 171s. Last read position: chr5:116,131,362
INFO 2018-10-13 16:09:48 SortSam Wrote 110,000,000 records from a sorting collection. Elapsed time: 00:56:23s. Time for last 10,000,000: 147s. Last read position: chr5:146,301,587
INFO 2018-10-13 16:12:06 SortSam Wrote 120,000,000 records from a sorting collection. Elapsed time: 00:58:41s. Time for last 10,000,000: 138s. Last read position: chr6:91,787,965
INFO 2018-10-13 16:14:47 SortSam Wrote 130,000,000 records from a sorting collection. Elapsed time: 01:01:22s. Time for last 10,000,000: 160s. Last read position: chr7:7,958,228
INFO 2018-10-13 16:17:14 SortSam Wrote 140,000,000 records from a sorting collection. Elapsed time: 01:03:49s. Time for last 10,000,000: 146s. Last read position: chr7:30,313,770
INFO 2018-10-13 16:19:35 SortSam Wrote 150,000,000 records from a sorting collection. Elapsed time: 01:06:09s. Time for last 10,000,000: 140s. Last read position: chr7:73,409,547
INFO 2018-10-13 16:21:52 SortSam Wrote 160,000,000 records from a sorting collection. Elapsed time: 01:08:27s. Time for last 10,000,000: 137s. Last read position: chr7:127,893,272
INFO 2018-10-13 16:24:04 SortSam Wrote 170,000,000 records from a sorting collection. Elapsed time: 01:10:39s. Time for last 10,000,000: 132s. Last read position: chr8:47,261,717
INFO 2018-10-13 16:26:07 SortSam Wrote 180,000,000 records from a sorting collection. Elapsed time: 01:12:42s. Time for last 10,000,000: 122s. Last read position: chr8:110,223,053
INFO 2018-10-13 16:28:40 SortSam Wrote 190,000,000 records from a sorting collection. Elapsed time: 01:15:14s. Time for last 10,000,000: 152s. Last read position: chr9:45,952,847
INFO 2018-10-13 16:31:16 SortSam Wrote 200,000,000 records from a sorting collection. Elapsed time: 01:17:51s. Time for last 10,000,000: 156s. Last read position: chr9:110,088,056
INFO 2018-10-13 16:33:58 SortSam Wrote 210,000,000 records from a sorting collection. Elapsed time: 01:20:33s. Time for last 10,000,000: 162s. Last read position: chr10:77,063,940
INFO 2018-10-13 16:36:26 SortSam Wrote 220,000,000 records from a sorting collection. Elapsed time: 01:23:01s. Time for last 10,000,000: 148s. Last read position: chr11:3,129,453
INFO 2018-10-13 16:39:11 SortSam Wrote 230,000,000 records from a sorting collection. Elapsed time: 01:25:46s. Time for last 10,000,000: 165s. Last read position: chr11:61,069,802
INFO 2018-10-13 16:41:58 SortSam Wrote 240,000,000 records from a sorting collection. Elapsed time: 01:28:33s. Time for last 10,000,000: 166s. Last read position: chr11:97,551,875
INFO 2018-10-13 16:44:41 SortSam Wrote 250,000,000 records from a sorting collection. Elapsed time: 01:31:16s. Time for last 10,000,000: 162s. Last read position: chr11:120,654,275
INFO 2018-10-13 16:47:26 SortSam Wrote 260,000,000 records from a sorting collection. Elapsed time: 01:34:01s. Time for last 10,000,000: 165s. Last read position: chr12:105,604,274
INFO 2018-10-13 16:50:16 SortSam Wrote 270,000,000 records from a sorting collection. Elapsed time: 01:36:51s. Time for last 10,000,000: 169s. Last read position: chr13:67,752,334
INFO 2018-10-13 16:53:15 SortSam Wrote 280,000,000 records from a sorting collection. Elapsed time: 01:39:50s. Time for last 10,000,000: 179s. Last read position: chr14:42,766,507
INFO 2018-10-13 16:56:11 SortSam Wrote 290,000,000 records from a sorting collection. Elapsed time: 01:42:45s. Time for last 10,000,000: 175s. Last read position: chr15:31,297,574
INFO 2018-10-13 16:58:56 SortSam Wrote 300,000,000 records from a sorting collection. Elapsed time: 01:45:31s. Time for last 10,000,000: 165s. Last read position: chr15:93,254,239
INFO 2018-10-13 17:01:50 SortSam Wrote 310,000,000 records from a sorting collection. Elapsed time: 01:48:24s. Time for last 10,000,000: 173s. Last read position: chr16:43,737,597
INFO 2018-10-13 17:04:37 SortSam Wrote 320,000,000 records from a sorting collection. Elapsed time: 01:51:12s. Time for last 10,000,000: 167s. Last read position: chr17:28,746,221
INFO 2018-10-13 17:07:21 SortSam Wrote 330,000,000 records from a sorting collection. Elapsed time: 01:53:56s. Time for last 10,000,000: 164s. Last read position: chr17:56,864,466
INFO 2018-10-13 17:10:21 SortSam Wrote 340,000,000 records from a sorting collection. Elapsed time: 01:56:56s. Time for last 10,000,000: 180s. Last read position: chr18:63,778,214
INFO 2018-10-13 17:13:09 SortSam Wrote 350,000,000 records from a sorting collection. Elapsed time: 01:59:43s. Time for last 10,000,000: 167s. Last read position: chr19:40,569,012
INFO 2018-10-13 17:15:56 SortSam Wrote 360,000,000 records from a sorting collection. Elapsed time: 02:02:31s. Time for last 10,000,000: 167s. Last read position: chrM:14,128
INFO 2018-10-13 17:18:37 SortSam Wrote 370,000,000 records from a sorting collection. Elapsed time: 02:05:12s. Time for last 10,000,000: 161s. Last read position: */*
[Sat Oct 13 17:18:53 UTC 2018] picard.sam.SortSam done. Elapsed time: 125.50 minutes.
Runtime.totalMemory()=3396337664
Log 4
[diop@fatnode1 Sample_9_B_MEF_Dox_TEST]$ /cm/shared/bioinfo/java/1.8.0_181/bin/java -jar /data/diop/picard.jar SortSam CREATE_INDEX=true INPUT=./Sample_9_B_MEF_Dox_TEST.3.bam OUTPUT=./Sample_9_B_MEF_Dox_TEST.3.sorted.bam SORT_ORDER=coordinate VALIDATION_STRINGENCY=LENIENT TMP_DIR=./tmp_dir3 INFO 2018-10-13 19:47:22 SortSam
********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
********** SortSam -CREATE_INDEX true -INPUT ./Sample_9_B_MEF_Dox_TEST.3.bam -OUTPUT ./Sample_9_B_MEF_Dox_TEST.3.sorted.bam -SORT_ORDER coordinate -VALIDATION_STRINGENCY LENIENT -TMP_DIR ./tmp_dir3
**********
19:47:24.081 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/data/diop/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Sat Oct 13 19:47:24 CEST 2018] SortSam INPUT=./Sample_9_B_MEF_Dox_TEST.3.bam OUTPUT=./Sample_9_B_MEF_Dox_TEST.3.sorted.bam SORT_ORDER=coordinate TMP_DIR=[./tmp_dir3] VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Sat Oct 13 19:47:24 CEST 2018] Executing as diop@fatnode1 on Linux 3.10.0-229.4.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_181-b13; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: unspecified-SNAPSHOT
INFO 2018-10-13 19:49:00 SortSam Read 10,000,000 records. Elapsed time: 00:01:35s. Time for last 10,000,000: 95s. Last read position: chr15:88,863,504
INFO 2018-10-13 19:50:28 SortSam Read 20,000,000 records. Elapsed time: 00:03:03s. Time for last 10,000,000: 87s. Last read position: chr8:104,617,748
INFO 2018-10-13 19:51:54 SortSam Read 30,000,000 records. Elapsed time: 00:04:29s. Time for last 10,000,000: 86s. Last read position: chr9:121,965,993
INFO 2018-10-13 19:53:11 SortSam Read 40,000,000 records. Elapsed time: 00:05:46s. Time for last 10,000,000: 76s. Last read position: chrM:3,603
INFO 2018-10-13 19:54:36 SortSam Read 50,000,000 records. Elapsed time: 00:07:11s. Time for last 10,000,000: 85s. Last read position: chr4:156,047,399
INFO 2018-10-13 19:56:05 SortSam Read 60,000,000 records. Elapsed time: 00:08:40s. Time for last 10,000,000: 88s. Last read position: chr8:120,160,321
INFO 2018-10-13 19:57:25 SortSam Read 70,000,000 records. Elapsed time: 00:10:00s. Time for last 10,000,000: 80s. Last read position: chr10:116,113,279
INFO 2018-10-13 19:58:47 SortSam Read 80,000,000 records. Elapsed time: 00:11:22s. Time for last 10,000,000: 82s. Last read position: chr17:34,650,291
INFO 2018-10-13 20:00:15 SortSam Read 90,000,000 records. Elapsed time: 00:12:51s. Time for last 10,000,000: 88s. Last read position: chr4:154,152,546
INFO 2018-10-13 20:01:38 SortSam Read 100,000,000 records. Elapsed time: 00:14:13s. Time for last 10,000,000: 82s. Last read position: chr4:131,780,062
INFO 2018-10-13 20:03:37 SortSam Read 110,000,000 records. Elapsed time: 00:16:12s. Time for last 10,000,000: 118s. Last read position: chr14:4,350,723
INFO 2018-10-13 20:05:06 SortSam Read 120,000,000 records. Elapsed time: 00:17:41s. Time for last 10,000,000: 89s. Last read position: chr7:98,492,472
INFO 2018-10-13 20:06:29 SortSam Read 130,000,000 records. Elapsed time: 00:19:04s. Time for last 10,000,000: 82s. Last read position: chr14:50,737,024
INFO 2018-10-13 20:07:55 SortSam Read 140,000,000 records. Elapsed time: 00:20:31s. Time for last 10,000,000: 86s. Last read position: chr4:42,850,249
INFO 2018-10-13 20:09:22 SortSam Read 150,000,000 records. Elapsed time: 00:21:57s. Time for last 10,000,000: 86s. Last read position: chr9:46,277,499
INFO 2018-10-13 20:10:53 SortSam Read 160,000,000 records. Elapsed time: 00:23:29s. Time for last 10,000,000: 91s. Last read position: chr7:58,813,631
INFO 2018-10-13 20:12:15 SortSam Read 170,000,000 records. Elapsed time: 00:24:50s. Time for last 10,000,000: 81s. Last read position: chr2:181,092,150
INFO 2018-10-13 20:13:40 SortSam Read 180,000,000 records. Elapsed time: 00:26:15s. Time for last 10,000,000: 84s. Last read position: chrX:20,637,486
INFO 2018-10-13 20:15:15 SortSam Read 190,000,000 records. Elapsed time: 00:27:50s. Time for last 10,000,000: 95s. Last read position: chr2:11,409,170
INFO 2018-10-13 20:16:42 SortSam Read 200,000,000 records. Elapsed time: 00:29:17s. Time for last 10,000,000: 87s. Last read position: chr4:135,465,627
INFO 2018-10-13 20:17:54 SortSam Read 210,000,000 records. Elapsed time: 00:30:29s. Time for last 10,000,000: 71s. Last read position: chr5:136,222,008
INFO 2018-10-13 20:18:59 SortSam Read 220,000,000 records. Elapsed time: 00:31:34s. Time for last 10,000,000: 65s. Last read position: chr9:29,411,744
INFO 2018-10-13 20:20:08 SortSam Read 230,000,000 records. Elapsed time: 00:32:44s. Time for last 10,000,000: 69s. Last read position: chr15:89,557,558
INFO 2018-10-13 20:21:18 SortSam Read 240,000,000 records. Elapsed time: 00:33:53s. Time for last 10,000,000: 69s. Last read position: chr2:155,709,064
INFO 2018-10-13 20:22:21 SortSam Read 250,000,000 records. Elapsed time: 00:34:56s. Time for last 10,000,000: 63s. Last read position: chr9:20,941,657
INFO 2018-10-13 20:23:29 SortSam Read 260,000,000 records. Elapsed time: 00:36:04s. Time for last 10,000,000: 67s. Last read position: chr13:88,821,859
INFO 2018-10-13 20:24:32 SortSam Read 270,000,000 records. Elapsed time: 00:37:07s. Time for last 10,000,000: 63s. Last read position: chr1:93,105,210
INFO 2018-10-13 20:25:45 SortSam Read 280,000,000 records. Elapsed time: 00:38:20s. Time for last 10,000,000: 72s. Last read position: chr10:76,581,546
INFO 2018-10-13 20:26:53 SortSam Read 290,000,000 records. Elapsed time: 00:39:28s. Time for last 10,000,000: 67s. Last read position: chr7:141,699,617
INFO 2018-10-13 20:27:59 SortSam Read 300,000,000 records. Elapsed time: 00:40:34s. Time for last 10,000,000: 66s. Last read position: chr9:83,688,474
INFO 2018-10-13 20:29:13 SortSam Read 310,000,000 records. Elapsed time: 00:41:49s. Time for last 10,000,000: 74s. Last read position: chr13:38,955,920
INFO 2018-10-13 20:30:21 SortSam Read 320,000,000 records. Elapsed time: 00:42:57s. Time for last 10,000,000: 68s. Last read position: chr2:150,586,385
INFO 2018-10-13 20:31:25 SortSam Read 330,000,000 records. Elapsed time: 00:44:01s. Time for last 10,000,000: 64s. Last read position: chr9:108,362,837
INFO 2018-10-13 20:32:30 SortSam Read 340,000,000 records. Elapsed time: 00:45:05s. Time for last 10,000,000: 64s. Last read position: chr2:98,666,818
INFO 2018-10-13 20:33:44 SortSam Read 350,000,000 records. Elapsed time: 00:46:19s. Time for last 10,000,000: 73s. Last read position: chr12:115,862,030
INFO 2018-10-13 20:34:55 SortSam Read 360,000,000 records. Elapsed time: 00:47:30s. Time for last 10,000,000: 71s. Last read position: chr19:5,570,366
INFO 2018-10-13 20:36:06 SortSam Read 370,000,000 records. Elapsed time: 00:48:42s. Time for last 10,000,000: 71s. Last read position: chr15:28,247,498
INFO 2018-10-13 20:36:11 SortSam Finished reading inputs, merging and writing to output now.
INFO 2018-10-13 20:36:12 SortingCollection Creating merging iterator from 742 files
INFO 2018-10-13 20:40:02 SortSam Wrote 10,000,000 records from a sorting collection. Elapsed time: 00:52:38s. Time for last 10,000,000: 176s. Last read position: chr1:87,389,279
INFO 2018-10-13 20:43:04 SortSam Wrote 20,000,000 records from a sorting collection. Elapsed time: 00:55:39s. Time for last 10,000,000: 181s. Last read position: chr1:180,935,153
INFO 2018-10-13 20:46:12 SortSam Wrote 30,000,000 records from a sorting collection. Elapsed time: 00:58:47s. Time for last 10,000,000: 187s. Last read position: chr2:60,422,678
INFO 2018-10-13 20:49:54 SortSam Wrote 40,000,000 records from a sorting collection. Elapsed time: 01:02:30s. Time for last 10,000,000: 222s. Last read position: chr2:145,934,434
INFO 2018-10-13 20:53:09 SortSam Wrote 50,000,000 records from a sorting collection. Elapsed time: 01:05:45s. Time for last 10,000,000: 195s. Last read position: chr3:36,671,235
INFO 2018-10-13 20:56:19 SortSam Wrote 60,000,000 records from a sorting collection. Elapsed time: 01:08:54s. Time for last 10,000,000: 189s. Last read position: chr3:126,842,587
INFO 2018-10-13 21:00:10 SortSam Wrote 70,000,000 records from a sorting collection. Elapsed time: 01:12:45s. Time for last 10,000,000: 230s. Last read position: chr4:101,526,887
INFO 2018-10-13 21:03:24 SortSam Wrote 80,000,000 records from a sorting collection. Elapsed time: 01:15:59s. Time for last 10,000,000: 193s. Last read position: chr4:140,758,199
INFO 2018-10-13 21:06:27 SortSam Wrote 90,000,000 records from a sorting collection. Elapsed time: 01:19:03s. Time for last 10,000,000: 183s. Last read position: chr5:30,934,373
INFO 2018-10-13 21:09:46 SortSam Wrote 100,000,000 records from a sorting collection. Elapsed time: 01:22:21s. Time for last 10,000,000: 198s. Last read position: chr5:116,131,362
INFO 2018-10-13 21:12:38 SortSam Wrote 110,000,000 records from a sorting collection. Elapsed time: 01:25:13s. Time for last 10,000,000: 172s. Last read position: chr5:146,301,587
INFO 2018-10-13 21:15:36 SortSam Wrote 120,000,000 records from a sorting collection. Elapsed time: 01:28:11s. Time for last 10,000,000: 178s. Last read position: chr6:91,787,965
INFO 2018-10-13 21:19:00 SortSam Wrote 130,000,000 records from a sorting collection. Elapsed time: 01:31:36s. Time for last 10,000,000: 204s. Last read position: chr7:7,958,228
INFO 2018-10-13 21:22:13 SortSam Wrote 140,000,000 records from a sorting collection. Elapsed time: 01:34:48s. Time for last 10,000,000: 192s. Last read position: chr7:30,313,770
INFO 2018-10-13 21:25:12 SortSam Wrote 150,000,000 records from a sorting collection. Elapsed time: 01:37:47s. Time for last 10,000,000: 178s. Last read position: chr7:73,409,547
INFO 2018-10-13 21:28:20 SortSam Wrote 160,000,000 records from a sorting collection. Elapsed time: 01:40:55s. Time for last 10,000,000: 188s. Last read position: chr7:127,893,272
INFO 2018-10-13 21:31:44 SortSam Wrote 170,000,000 records from a sorting collection. Elapsed time: 01:44:20s. Time for last 10,000,000: 204s. Last read position: chr8:47,261,717
INFO 2018-10-13 21:34:49 SortSam Wrote 180,000,000 records from a sorting collection. Elapsed time: 01:47:24s. Time for last 10,000,000: 184s. Last read position: chr8:110,223,053
INFO 2018-10-13 21:37:53 SortSam Wrote 190,000,000 records from a sorting collection. Elapsed time: 01:50:28s. Time for last 10,000,000: 184s. Last read position: chr9:45,952,847
INFO 2018-10-13 21:41:03 SortSam Wrote 200,000,000 records from a sorting collection. Elapsed time: 01:53:38s. Time for last 10,000,000: 190s. Last read position: chr9:110,088,056
INFO 2018-10-13 21:44:06 SortSam Wrote 210,000,000 records from a sorting collection. Elapsed time: 01:56:42s. Time for last 10,000,000: 183s. Last read position: chr10:77,063,940
INFO 2018-10-13 21:46:59 SortSam Wrote 220,000,000 records from a sorting collection. Elapsed time: 01:59:35s. Time for last 10,000,000: 173s. Last read position: chr11:3,129,453
INFO 2018-10-13 21:50:18 SortSam Wrote 230,000,000 records from a sorting collection. Elapsed time: 02:02:54s. Time for last 10,000,000: 198s. Last read position: chr11:61,069,802
INFO 2018-10-13 21:53:43 SortSam Wrote 240,000,000 records from a sorting collection. Elapsed time: 02:06:18s. Time for last 10,000,000: 204s. Last read position: chr11:97,551,875
INFO 2018-10-13 21:57:16 SortSam Wrote 250,000,000 records from a sorting collection. Elapsed time: 02:09:51s. Time for last 10,000,000: 212s. Last read position: chr11:120,654,275
INFO 2018-10-13 22:01:09 SortSam Wrote 260,000,000 records from a sorting collection. Elapsed time: 02:13:44s. Time for last 10,000,000: 233s. Last read position: chr12:105,604,274
INFO 2018-10-13 22:04:47 SortSam Wrote 270,000,000 records from a sorting collection. Elapsed time: 02:17:22s. Time for last 10,000,000: 217s. Last read position: chr13:67,752,334
INFO 2018-10-13 22:08:15 SortSam Wrote 280,000,000 records from a sorting collection. Elapsed time: 02:20:50s. Time for last 10,000,000: 208s. Last read position: chr14:42,766,507
INFO 2018-10-13 22:11:42 SortSam Wrote 290,000,000 records from a sorting collection. Elapsed time: 02:24:17s. Time for last 10,000,000: 206s. Last read position: chr15:31,297,574
INFO 2018-10-13 22:15:22 SortSam Wrote 300,000,000 records from a sorting collection. Elapsed time: 02:27:57s. Time for last 10,000,000: 219s. Last read position: chr15:93,254,239
INFO 2018-10-13 22:18:46 SortSam Wrote 310,000,000 records from a sorting collection. Elapsed time: 02:31:21s. Time for last 10,000,000: 204s. Last read position: chr16:43,737,597
INFO 2018-10-13 22:22:40 SortSam Wrote 320,000,000 records from a sorting collection. Elapsed time: 02:35:15s. Time for last 10,000,000: 233s. Last read position: chr17:28,746,221
INFO 2018-10-13 22:25:42 SortSam Wrote 330,000,000 records from a sorting collection. Elapsed time: 02:38:17s. Time for last 10,000,000: 182s. Last read position: chr17:56,864,466
INFO 2018-10-13 22:29:22 SortSam Wrote 340,000,000 records from a sorting collection. Elapsed time: 02:41:57s. Time for last 10,000,000: 219s. Last read position: chr18:63,778,214
[Sat Oct 13 22:29:54 CEST 2018] picard.sam.SortSam done. Elapsed time: 162.51 minutes.
Runtime.totalMemory()=3310354432
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: Read error; BinaryCodec in readmode; streamed file (filename not available)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:420)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:394)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:380)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:268)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:40)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.advance(SortingCollection.java:626)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.next(SortingCollection.java:616)
at htsjdk.samtools.util.PeekIterator.peek(PeekIterator.java:69)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:653)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:648)
at java.util.TreeMap.put(TreeMap.java:552)
at java.util.TreeSet.add(TreeSet.java:255)
at htsjdk.samtools.util.SortingCollection$MergingIterator.next(SortingCollection.java:562)
at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:223)
at picard.sam.SortSam.doWork(SortSam.java:167)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:295)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:98)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:474)
at org.xerial.snappy.Snappy.uncompress(Snappy.java:513)
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:439)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:167)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:418)
... 17 more
Log 5
diop@fatnode1 ~]$ /cm/shared/bioinfo/java/1.8.0_181/bin/java -Dsnappy.disable=true -jar /cm/shared/bioinfo/picard/1.134/picard.jar SortSam CREATE_INDEX=true INPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam OUTPUT=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam SORT_ORDER=coordinate VALIDATION_STRINGENCY=LENIENT TMP_DIR=/data/diop/B18_BELO2/all_runs//Sample_9_B_MEF_Dox_TEST/tmp_dir
[Sun Oct 14 01:36:47 CEST 2018] picard.sam.SortSam INPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.bam OUTPUT=/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/Sample_9_B_MEF_Dox_TEST.sorted.bam SORT_ORDER=coordinate TMP_DIR=[/data/diop/B18_BELO2/all_runs/Sample_9_B_MEF_Dox_TEST/tmp_dir] VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
[Sun Oct 14 01:36:47 CEST 2018] Executing as diop@fatnode1 on Linux 3.10.0-229.4.2.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_181-b13; Picard version: 1.134(d8c82d895da53bc5930aa36974f0b9f11df1e74b_1434985389) JdkDeflater
Snappy is disabled via system property.
INFO 2018-10-14 01:37:57 SortSam Read 10,000,000 records. Elapsed time: 00:01:09s. Time for last 10,000,000: 69s. Last read position: chr15:88,863,504
INFO 2018-10-14 01:39:02 SortSam Read 20,000,000 records. Elapsed time: 00:02:13s. Time for last 10,000,000: 64s. Last read position: chr8:104,617,748
INFO 2018-10-14 01:40:17 SortSam Read 30,000,000 records. Elapsed time: 00:03:29s. Time for last 10,000,000: 75s. Last read position: chr9:121,965,993
INFO 2018-10-14 01:41:33 SortSam Read 40,000,000 records. Elapsed time: 00:04:45s. Time for last 10,000,000: 76s. Last read position: chrM:3,603
INFO 2018-10-14 01:42:41 SortSam Read 50,000,000 records. Elapsed time: 00:05:53s. Time for last 10,000,000: 68s. Last read position: chr4:156,047,399
INFO 2018-10-14 01:43:48 SortSam Read 60,000,000 records. Elapsed time: 00:07:00s. Time for last 10,000,000: 66s. Last read position: chr8:120,160,321
INFO 2018-10-14 01:45:05 SortSam Read 70,000,000 records. Elapsed time: 00:08:17s. Time for last 10,000,000: 77s. Last read position: chr10:116,113,279
INFO 2018-10-14 01:46:06 SortSam Read 80,000,000 records. Elapsed time: 00:09:18s. Time for last 10,000,000: 61s. Last read position: chr17:34,650,291
INFO 2018-10-14 01:47:22 SortSam Read 90,000,000 records. Elapsed time: 00:10:34s. Time for last 10,000,000: 75s. Last read position: chr4:154,152,546
INFO 2018-10-14 01:48:55 SortSam Read 100,000,000 records. Elapsed time: 00:12:07s. Time for last 10,000,000: 92s. Last read position: chr4:131,780,062
INFO 2018-10-14 01:50:08 SortSam Read 110,000,000 records. Elapsed time: 00:13:20s. Time for last 10,000,000: 72s. Last read position: chr14:4,350,723
INFO 2018-10-14 01:51:15 SortSam Read 120,000,000 records. Elapsed time: 00:14:27s. Time for last 10,000,000: 66s. Last read position: chr7:98,492,472
INFO 2018-10-14 01:52:23 SortSam Read 130,000,000 records. Elapsed time: 00:15:35s. Time for last 10,000,000: 68s. Last read position: chr14:50,737,024
INFO 2018-10-14 01:53:26 SortSam Read 140,000,000 records. Elapsed time: 00:16:38s. Time for last 10,000,000: 62s. Last read position: chr4:42,850,249
INFO 2018-10-14 01:54:35 SortSam Read 150,000,000 records. Elapsed time: 00:17:47s. Time for last 10,000,000: 69s. Last read position: chr9:46,277,499
INFO 2018-10-14 01:55:34 SortSam Read 160,000,000 records. Elapsed time: 00:18:46s. Time for last 10,000,000: 58s. Last read position: chr7:58,813,631
INFO 2018-10-14 01:56:34 SortSam Read 170,000,000 records. Elapsed time: 00:19:45s. Time for last 10,000,000: 59s. Last read position: chr2:181,092,150
INFO 2018-10-14 01:57:33 SortSam Read 180,000,000 records. Elapsed time: 00:20:45s. Time for last 10,000,000: 59s. Last read position: chrX:20,637,486
INFO 2018-10-14 01:58:38 SortSam Read 190,000,000 records. Elapsed time: 00:21:50s. Time for last 10,000,000: 64s. Last read position: chr2:11,409,170
INFO 2018-10-14 01:59:35 SortSam Read 200,000,000 records. Elapsed time: 00:22:47s. Time for last 10,000,000: 57s. Last read position: chr4:135,465,627
INFO 2018-10-14 02:00:45 SortSam Read 210,000,000 records. Elapsed time: 00:23:57s. Time for last 10,000,000: 69s. Last read position: chr5:136,222,008
INFO 2018-10-14 02:01:58 SortSam Read 220,000,000 records. Elapsed time: 00:25:10s. Time for last 10,000,000: 72s. Last read position: chr9:29,411,744
INFO 2018-10-14 02:02:57 SortSam Read 230,000,000 records. Elapsed time: 00:26:09s. Time for last 10,000,000: 59s. Last read position: chr15:89,557,558
INFO 2018-10-14 02:03:48 SortSam Read 240,000,000 records. Elapsed time: 00:27:00s. Time for last 10,000,000: 51s. Last read position: chr2:155,709,064
INFO 2018-10-14 02:04:45 SortSam Read 250,000,000 records. Elapsed time: 00:27:57s. Time for last 10,000,000: 56s. Last read position: chr9:20,941,657
INFO 2018-10-14 02:05:50 SortSam Read 260,000,000 records. Elapsed time: 00:29:02s. Time for last 10,000,000: 65s. Last read position: chr13:88,821,859
INFO 2018-10-14 02:06:51 SortSam Read 270,000,000 records. Elapsed time: 00:30:03s. Time for last 10,000,000: 60s. Last read position: chr1:93,105,210
INFO 2018-10-14 02:07:59 SortSam Read 280,000,000 records. Elapsed time: 00:31:10s. Time for last 10,000,000: 67s. Last read position: chr10:76,581,546
INFO 2018-10-14 02:08:55 SortSam Read 290,000,000 records. Elapsed time: 00:32:07s. Time for last 10,000,000: 56s. Last read position: chr7:141,699,617
INFO 2018-10-14 02:09:56 SortSam Read 300,000,000 records. Elapsed time: 00:33:08s. Time for last 10,000,000: 61s. Last read position: chr9:83,688,474
INFO 2018-10-14 02:10:58 SortSam Read 310,000,000 records. Elapsed time: 00:34:10s. Time for last 10,000,000: 62s. Last read position: chr13:38,955,920
INFO 2018-10-14 02:12:10 SortSam Read 320,000,000 records. Elapsed time: 00:35:22s. Time for last 10,000,000: 71s. Last read position: chr2:150,586,385
INFO 2018-10-14 02:13:18 SortSam Read 330,000,000 records. Elapsed time: 00:36:30s. Time for last 10,000,000: 68s. Last read position: chr9:108,362,837
INFO 2018-10-14 02:14:22 SortSam Read 340,000,000 records. Elapsed time: 00:37:34s. Time for last 10,000,000: 63s. Last read position: chr2:98,666,818
INFO 2018-10-14 02:15:23 SortSam Read 350,000,000 records. Elapsed time: 00:38:35s. Time for last 10,000,000: 61s. Last read position: chr12:115,862,030
INFO 2018-10-14 02:16:41 SortSam Read 360,000,000 records. Elapsed time: 00:39:52s. Time for last 10,000,000: 77s. Last read position: chr19:5,570,366
INFO 2018-10-14 02:17:41 SortSam Read 370,000,000 records. Elapsed time: 00:40:53s. Time for last 10,000,000: 60s. Last read position: chr15:28,247,498
INFO 2018-10-14 02:17:49 SortSam Finished reading inputs, merging and writing to output now.
INFO 2018-10-14 02:24:53 SortSam Wrote 10,000,000 records from a sorting collection. Elapsed time: 00:48:05s. Time for last 10,000,000: 376s. Last read position: chr1:87,389,279
INFO 2018-10-14 02:31:00 SortSam Wrote 20,000,000 records from a sorting collection. Elapsed time: 00:54:12s. Time for last 10,000,000: 367s. Last read position: chr1:180,935,153
INFO 2018-10-14 02:37:07 SortSam Wrote 30,000,000 records from a sorting collection. Elapsed time: 01:00:19s. Time for last 10,000,000: 366s. Last read position: chr2:60,422,678
INFO 2018-10-14 02:43:22 SortSam Wrote 40,000,000 records from a sorting collection. Elapsed time: 01:06:34s. Time for last 10,000,000: 375s. Last read position: chr2:145,934,434
INFO 2018-10-14 02:49:51 SortSam Wrote 50,000,000 records from a sorting collection. Elapsed time: 01:13:03s. Time for last 10,000,000: 389s. Last read position: chr3:36,671,235
INFO 2018-10-14 02:57:04 SortSam Wrote 60,000,000 records from a sorting collection. Elapsed time: 01:20:16s. Time for last 10,000,000: 432s. Last read position: chr3:126,842,587
INFO 2018-10-14 03:03:18 SortSam Wrote 70,000,000 records from a sorting collection. Elapsed time: 01:26:30s. Time for last 10,000,000: 373s. Last read position: chr4:101,526,887
INFO 2018-10-14 03:09:59 SortSam Wrote 80,000,000 records from a sorting collection. Elapsed time: 01:33:10s. Time for last 10,000,000: 400s. Last read position: chr4:140,758,199
INFO 2018-10-14 03:16:39 SortSam Wrote 90,000,000 records from a sorting collection. Elapsed time: 01:39:51s. Time for last 10,000,000: 400s. Last read position: chr5:30,934,373
INFO 2018-10-14 03:23:35 SortSam Wrote 100,000,000 records from a sorting collection. Elapsed time: 01:46:47s. Time for last 10,000,000: 416s. Last read position: chr5:116,131,362
INFO 2018-10-14 03:30:23 SortSam Wrote 110,000,000 records from a sorting collection. Elapsed time: 01:53:35s. Time for last 10,000,000: 408s. Last read position: chr5:146,301,587
INFO 2018-10-14 03:37:24 SortSam Wrote 120,000,000 records from a sorting collection. Elapsed time: 02:00:36s. Time for last 10,000,000: 421s. Last read position: chr6:91,787,965
INFO 2018-10-14 03:45:09 SortSam Wrote 130,000,000 records from a sorting collection. Elapsed time: 02:08:20s. Time for last 10,000,000: 464s. Last read position: chr7:7,958,228
INFO 2018-10-14 03:52:24 SortSam Wrote 140,000,000 records from a sorting collection. Elapsed time: 02:15:36s. Time for last 10,000,000: 435s. Last read position: chr7:30,313,770
[Sun Oct 14 03:54:16 CEST 2018] picard.sam.SortSam done. Elapsed time: 137.48 minutes.
Runtime.totalMemory()=4731699200
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.SAMFormatException: Invalid record length: 0
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:184)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:37)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.advance(SortingCollection.java:484)
at htsjdk.samtools.util.SortingCollection$FileRecordIterator.next(SortingCollection.java:475)
at htsjdk.samtools.util.PeekIterator.peek(PeekIterator.java:67)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:507)
at htsjdk.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:504)
at java.util.TreeMap.put(TreeMap.java:552)
at java.util.TreeSet.add(TreeSet.java:255)
at htsjdk.samtools.util.SortingCollection$MergingIterator.next(SortingCollection.java:423)
at htsjdk.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:203)
at picard.sam.SortSam.doWork(SortSam.java:89)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:206)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
The issue happens with at least the two different versions of SortSam. Sometimes it says “Failed to Uncompress”, sometimes it’s “Parsing Error”. Always comming from SnappyNative. I’ve tried a lot of stuff, such as:
- rebuilding Picard with the latest htsjdk (wouldn’t fix, but got a speed improvement)
- upgrading my system
- using the system’s snappy
- putting my bam file on a different filesystem
- using different compute nodes
- using different users
- using singularity’s picard …
I’ve not included every log (actually, only have the last ones, didn’t think of saving them since they all look alike).
Steps to reproduce
Well, that might be complex, I do not know. Maybe I could give my BAM file out to you ? You’ll notice the error always takes place in the writing step, but the writing steps sometimes gets interrupted “before end” (according the read counter) by the error. Sometimes, it just works.
Expected behavior
Well, sort the sam and end well
Issue Analytics
- State:
- Created 5 years ago
- Comments:8 (4 by maintainers)
Top Results From Across the Web
8070. A problem of Picard - Legacy GATK Forum - Google Sites
I am newbie of GATK.I have met a problem when I use picad Sortsam. ... So I need your help. ... This is...
Read more >Frequently Asked Questions - GitHub Pages
Q: A Picard tool that sorts its output SAM/BAM file is taking a very long time and/or running ... It says In arrows(metrics$GC,...
Read more >Thread: [Samtools-help] error using samsort | SAM tools
One thing that occurs to me is that Picard uses a C library, snappy, to compress the temporary files used in sorting.
Read more >[HTCondor-users] issue with java(possibly) - cannot find non ...
[HTCondor-users] issue with java(possibly) - cannot find non-existent file ... /tools/Picard/picard/SortSam.jar, /tools/Picard/snappy-java-1.0.3-rc3.jar
Read more >Fatal error detected by the Java Runtime Environment ... - GATK
This issue is being filed on behalf of Eduardo Maury. ... Eduardo Maury it looks like when you tried Snappy, the WDL wasn't...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@devenderarora Sorry, I accidentally marked the issue as closed. I merged a newer version of snappy into htsjdk so if you know how to build picard with a snapshot htsjdk you could try that to see if it happens to fix whatever the bug is.
If you could give us the bam file that causes the crash I might be able to do more debugging, but without a way to reproduce it locally I’m not sure I can do much more.
Replying late, I was away and yes multiple java process running at one time I will give a try by setting it to true.