Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Error : Creating checkpoint file

See original GitHub issue

Hi,

I have been trying to run your pipeline with the mm10 data installed from install_genome_data. My output looks like something in the file “myOutStruct” compared to the structure given on the ATAC-seq gitHub page (screenshot in “originalOutStruct”)

I am running the command :

 $ bds atac.bds -species mm10 -gensz mm -bam1

…/genomes/mm10_enh_dhs.bam -chrsz …/genomes/mm10/mm10.chrom.sizes

I have generated the bam file using the bed files within the mm10 download/install.

It also generates the error file (“task.postalign_bam.markdup_bam_picard_rep1.line_409.id_10.stderr”). Which looks something like :

Picked up _JAVA_OPTIONS: -Xms256M -Xmx12G -XX:ParallelGCThreads=1 [Wed Jun 27 17:07:04 BST 2018] picard.sam.markduplicates.MarkDuplicates INPUT=[/homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.filt.bam] OUTPUT=/homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam METRICS_FILE=/homes/reham/ATAC-Seq/atac_dnase_pipelines/out/qc/rep1/mm10_enh_dhs.dup.qc REMOVE_DUPLICATES=false ASSUME_SORTED=true VALIDATION_STRINGENCY=LENIENT MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+)😦[0-9]+)😦[0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false [Wed Jun 27 17:07:04 BST 2018] Executing as reham@hx-noah-08-10.ebi.ac.uk on Linux 3.10.0-514.16.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_131-b11; Picard version: 1.126(4691ee611ac205d4afe2a1b7a2ea975a6f997426_1417447214) IntelDeflater INFO 2018-06-27 17:07:04 MarkDuplicates Start of doWork freeMemory: 256939704; totalMemory: 259588096; maxMemory: 12455444480 INFO 2018-06-27 17:07:04 MarkDuplicates Reading input file and constructing read end information. INFO 2018-06-27 17:07:04 MarkDuplicates Will retain up to 47905555 data points before spilling to disk. INFO 2018-06-27 17:07:07 MarkDuplicates Read 0 records. 0 pairs never matched. INFO 2018-06-27 17:07:07 MarkDuplicates After buildSortedReadEndLists freeMemory: 539996048; totalMemory: 929882112; maxMemory: 12455444480 INFO 2018-06-27 17:07:07 MarkDuplicates Will retain up to 389232640 duplicate indices before spilling to disk.

The output I am presently getting concludes with the following :

Distributing 8 to ... 
{1=8}
pyenv: activate: command not found
Task failed:
	Program & line     : 'modules/postalign_bam.bds', line 341
	Task Name          : 'dedup_bam_PE_2 rep1'
	Task ID            : 'atac.bds.20180628_151527_655_parallel_41/task.postalign_bam.dedup_bam_PE_2_rep1.line_341.id_10'
	Task PID           : '4097'
	Task hint          : 'samtools view -F 1804 -f 2 -b /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam > /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.nodup.bam; sambamba index -t 8 /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.nodup.bam; s'
	Task resources     : 'cpus: 8	mem: -1.0 B	wall-timeout: 8640000'
	State              : 'ERROR'
	Dependency state   : 'ERROR'
	Retries available  : '1'
	Input files        : '[/homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam]'
	Output files       : '[/homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.nodup.bam, /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/qc/rep1/mm10_enh_dhs.nodup.flagstat.qc, /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/qc/rep1/mm10_enh_dhs.nodup.pbc.qc]'
	Script file        : '/homes/reham/ATAC-Seq/atac_dnase_pipelines/atac.bds.20180628_151527_655_parallel_41/task.postalign_bam.dedup_bam_PE_2_rep1.line_341.id_10.sh'
	Exit status        : '1'
	Program            : 
		
		# SYS command. line 343
		
		 if [[ -f $(which conda) && $(conda env list | grep bds_atac | wc -l) != "0" ]]; then source activate bds_atac; sleep 5; fi;  export PATH=/homes/reham/ATAC-Seq/atac_dnase_pipelines/.:/homes/reham/ATAC-Seq/atac_dnase_pipelines/modules:/homes/reham/ATAC-Seq/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)
		
		# SYS command. line 350
		
		 samtools view -F 1804 -f 2 -b /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam > /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.nodup.bam
		
		# SYS command. line 352
		
		 sambamba index -t 8 /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.nodup.bam
		
		# SYS command. line 354
		
		 sambamba flagstat -t 8 /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.nodup.bam > /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/qc/rep1/mm10_enh_dhs.nodup.flagstat.qc
		
		# SYS command. line 365
		
		 sambamba sort -t 8 -n /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam -o /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam.tmp.bam
		
		# SYS command. line 367
		
		 bedtools bamtobed -bedpe -i /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam.tmp.bam | \
							awk 'BEGIN{OFS="\t"}{print $1,$2,$4,$6,$9,$10}' | \
							grep -v 'chrM' | sort | uniq -c | \
							awk 'BEGIN{mt=0;m0=0;m1=0;m2=0} ($1==1){m1=m1+1} ($1==2){m2=m2+1} {m0=m0+1} {mt=mt+$1} END{m1_m2=-1.0; if(m2>0) m1_m2=m1/m2; printf "%d\t%d\t%d\t%d\t%f\t%f\t%f\n",mt,m0,m1,m2,m0/mt,m1/m0,m1_m2}' > /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/qc/rep1/mm10_enh_dhs.nodup.pbc.qc
		
		# SYS command. line 371
		
		 rm -f /homes/reham/ATAC-Seq/atac_dnase_pipelines/out/align/rep1/mm10_enh_dhs.dupmark.bam.tmp.bam
		
		# SYS command. line 373
		
		 TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0
	StdErr (100000000 lines)  :
		pyenv: activate: command not found

Fatal error: modules/postalign_bam.bds, line 378, pos 4. Task/s failed.

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.
Fatal error: atac.bds, line 426, pos 2. Task/s failed.

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.

Could you please help me in generating the right output ?

Many thanks. Kind Regards, Reham