SamToFastq no longer writes to /dev/stdout so I cannot pipe

SamToFastq no longer writes to /dev/stdout so I cannot pipe it to bwa mem. I’m told this could be a regression and that I should file a bug report.

---

Bug Report

Affected tool(s)

Picard SamToFastq

Affected version(s)

• Latest public release version [version 2.8.2]

Description

SamToFastq no longer writes to /dev/stdout when FASTQ=/dev/stdout. I am working on the Broad servers. Previously, I got this piping to work on my laptop and come to think of it on the server too.

Expected behavior

Piping should work SamToFastq should write to /dev/stdout on the Broad servers.

Actual behavior

-bash-4.1$ java -Xmx8G -jar $PICARD SamToFastq I=hcc1143_tumor_u.bam FASTQ=/dev/stdout INTERLEAVE=true INCLUDE_NON_PF_READS=true | $bwa0715 -p /humgen/gsa-hpprojects/dev/shlee/ref/GRCh38_1kg/GRCh38_full_analysi
s_set_plus_decoy_hla.fa /dev/stdin > hcc1143_tumor.sam
[main] unrecognized command '-p'
[Fri Feb 03 17:33:57 EST 2017] picard.sam.SamToFastq INPUT=hcc1143_tumor_u.bam FASTQ=/dev/stdout INTERLEAVE=true INCLUDE_NON_PF_READS=true    OUTPUT_PER_RG=false RG_TAG=PU RE_REVERSE=true CLIPPING_MIN_LENGTH=0 
READ1_TRIM=0 READ2_TRIM=0 INCLUDE_NON_PRIMARY_ALIGNMENTS=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4G
H_CLIENT_SECRETS=client_secrets.json
[Fri Feb 03 17:33:57 EST 2017] Executing as [e-mail] on Linux 2.6.32-642.11.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13; Picard version: 2.8.2-SNAPSHOT
[Fri Feb 03 17:33:57 EST 2017] picard.sam.SamToFastq done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=2025848832
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.SAMException: Error in writing fastq file /dev/stdout
        at htsjdk.samtools.fastq.BasicFastqWriter.write(BasicFastqWriter.java:68)
        at picard.sam.SamToFastq.writeRecord(SamToFastq.java:350)
        at picard.sam.SamToFastq.doWork(SamToFastq.java:203)
        at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:205)
        at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:94)
        at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:104)

And

-bash-4.1$ ls -l /dev/stdout
lrwxrwxrwx 1 root root 15 Dec 16 06:02 /dev/stdout -> /proc/self/fd/1
-bash-4.1$ echo $PICARD
/seq/software/picard/current/bin/picard.jar
-bash-4.1$ 

[micknudsen]

@Pharah01 Did you figure out a solution to your problem? I am having the same issue.

[shuang-luo]

Dear gatk team, I met similar questions but I did´t lack mem. It seems like, my code cannot call bwa mem. But I didn’t know why and how to fix it. I use the same wdl offered by gatk GitHub which updated 2 month ago. https://github.com/gatk-workflows/gatk4-data-processing/blob/master/processing-for-variant-discovery-gatk4.wdl

my error report: apro13-36hv29:execution shuanglu$ more stderr Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/ac1b257a-b883-47c2-a124-c1dec5ca2c7f/call-SamToFastqAndBwaMem/shard-6/tmp.c2fea1ef 22:44:21.277 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/usr/gitc/picard.jar!/com/intel/gkl/native/libgkl_compression.so [Sun Aug 19 22:44:21 UTC 2018] SamToFastq INPUT=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/ac1b257a-b883-47c2-a124-c1dec5ca2c7f/call-SamToFastqAndBwaMem/shard-6/inputs/-1729239788/wgs_ubam-NA12878_24RG-small-HK3T5.1.NA12878.interval.filtered.query.sorted.unmapped.bam FASTQ=/dev/stdout INTERLEAVE=true INCLUDE_NON_PF_READS=true OUTPUT_PER_RG=false COMPRESS_OUTPUTS_PER_RG=false RG_TAG=PU RE_REVERSE=true CLIPPING_MIN_LENGTH=0 READ1_TRIM=0 READ2_TRIM=0 INCLUDE_NON_PRIMARY_ALIGNMENTS=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false [Sun Aug 19 22:44:21 UTC 2018] Executing as root@6c346e439f17 on Linux 4.9.93-linuxkit-aufs amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2~bpo8+1-b14; Deflater: Intel; Inflater: Intel; Picard version: 2.16.0-SNAPSHOT [Sun Aug 19 22:44:25 UTC 2018] picard.sam.SamToFastq done. Elapsed time: 0.07 minutes. Runtime.totalMemory()=3014656000 To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp Exception in thread “main” htsjdk.samtools.SAMException: Error in writing fastq file /dev/stdout at htsjdk.samtools.fastq.BasicFastqWriter.write(BasicFastqWriter.java:66) at picard.sam.SamToFastq.writeRecord(SamToFastq.java:356) at picard.sam.SamToFastq.doWork(SamToFastq.java:206) at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:268) at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98) at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108) /cromwell-executions/PreProcessingForVariantDiscovery_GATK4/ac1b257a-b883-47c2-a124-c1dec5ca2c7f/call-SamToFastqAndBwaMem/shard-6/execution/script: line 33: 11 Exit 1 java -Dsamjdk.compression_level=5 -Xms3000m -jar /usr/gitc/picard.jar SamToFastq INPUT=/cromwell-executions/PreProcessingForVariantDiscovery_GATK4/ac1b257a-b883-47c2-a124-c1dec5ca2c7f/call-SamToFastqAndBwaMem/shard-6/inputs/-1729239788/wgs_ubam-NA12878_24RG-small-HK3T5.1.NA12878.interval.filtered.query.sorted.unmapped.bam FASTQ=/dev/stdout INTERLEAVE=true NON_PF=true 12 Killed | /usr/gitc/bwa mem -K 100000000 -p -v 3 -t 16 -Y $bash_ref_fasta /dev/stdin - 2> >(tee wgs_ubam-NA12878_24RG-small-HK3T5.1.NA12878.interval.filtered.query.sorted.unmapped.unmerged.bwa.stderr.log >&2) 13 Done | samtools view -1 - > wgs_ubam-NA12878_24RG-small-HK3T5.1.NA12878.interval.filtered.query.sorted.unmapped.unmerged.bam

json file , I used: (possible relative part)

“##_COMMENT4”: “MISC PARAMETERS”, “PreProcessingForVariantDiscovery_GATK4.bwa_commandline”: “bwa mem -K 100000000 -p -v 3 -t 16 -Y $bash_ref_fasta”, “PreProcessingForVariantDiscovery_GATK4.compression_level”: 5, “PreProcessingForVariantDiscovery_GATK4.SamToFastqAndBwaMem.num_cpu”: “16”,

“##_COMMENT5”: “DOCKERS”, “PreProcessingForVariantDiscovery_GATK4.gotc_docker”: “broadinstitute/genomes-in-the-cloud:2.3.1-1512499786”, “PreProcessingForVariantDiscovery_GATK4.gatk_docker”: “broadinstitute/gatk:4.0.4.0”, “PreProcessingForVariantDiscovery_GATK4.picard_docker”: “broadinstitute/genomes-in-the-cloud:2.3.1-1512499786”, “PreProcessingForVariantDiscovery_GATK4.python_docker”: “python:2.7”,

“##_COMMENT6”: “PATHS”,
“PreProcessingForVariantDiscovery_GATK4.gotc_path”: “/usr/gitc/”, “PreProcessingForVariantDiscovery_GATK4.picard_path”:“/usr/gitc/”, “PreProcessingForVariantDiscovery_GATK4.gatk_path”: “/gatk/gatk”, “PreProcessingForVariantDiscovery_GATK4.bwa_path”:“/usr/gitc/”,