Compiled from source, tests pass but "make examples" does not run with test data.

Hi! I compiled from source on Ubuntu 14.4.

Tests pass, including ./bazel-bin/deepvariant/make_examples_test.

However, running as simple make_examples with downloaded example data fails with confusing (for me) error in realign script. Same script worked fine when I ran pre-compiled binary from the Docker container 🤷‍♂️

input="./quickstart-testdata"

./bazel-bin/deepvariant/make_examples \
  --ref=$input/ucsc.hg19.chr20.unittest.fasta \
  --reads=$input/NA12878_S1.chr20.10_10p1mb.bam \
  --regions "chr20:10,000,000-10,010,000" \
  --examples [e-mail] \
  --mode calling \
  --logging_every_n_candidates 10 \
  --realign_reads

./make_examples_demo.sh 
2019-07-16 17:49:02.877175: W third_party/nucleus/io/sam_reader.cc:564] Unrecognized SAM header type, ignoring: 
I0716 17:49:02.877284 139897470359360 genomics_reader.py:218] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I0716 17:49:03.117142 139897470359360 make_examples.py:1110] Preparing inputs
2019-07-16 17:49:03.117644: W third_party/nucleus/io/sam_reader.cc:564] Unrecognized SAM header type, ignoring: 
I0716 17:49:03.117749 139897470359360 genomics_reader.py:218] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I0716 17:49:03.118745 139897470359360 make_examples.py:1034] Common contigs are [u'chr20']
I0716 17:49:03.120177 139897470359360 make_examples.py:1116] Writing examples to examples.tfrecord-00000-of-00001.gz
2019-07-16 17:49:03.121118: I third_party/nucleus/io/sam_reader.cc:600] Setting HTS_OPT_BLOCK_SIZE to 134217728
2019-07-16 17:49:03.124279: W third_party/nucleus/io/sam_reader.cc:564] Unrecognized SAM header type, ignoring: 
I0716 17:49:03.124422 139897470359360 genomics_reader.py:218] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1235, in <module>
    tf.app.run()
  File "/home/nyakovenko/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1225, in main
    make_examples_runner(options)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 1127, in make_examples_runner
    candidates, examples, gvcfs = region_processor.process(region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 849, in process
    self.in_memory_sam_reader.replace_reads(self.region_reads(region))
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 889, in region_reads
    _, reads = self.realigner.realign_reads(reads, region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/realigner.py", line 606, in realign_reads
    self.config.ws_config, self.ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 232, in select_windows
    candidates = _candidates_from_reads(config, ref_reader, reads, region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 84, in _candidates_from_reads
    region, expanded_region)
  File "/tmp/Bazel.runfiles_yOE450/runfiles/com_google_deepvariant/deepvariant/realigner/window_selector.py", line 152, in _allele_count_linear_selector
    allele_counter, model_conf))
TypeError: allele_count_linear_candidates_from_allele_counter() argument counter is not valid for ::learning::genomics::deepvariant::AlleleCounter (deepvariant.python.allelecounter.AlleleCounter instance given): expecting deepvariant.python.allelecounter.AlleleCounter instance, got deepvariant.python.allelecounter.AlleleCounter instance

[pichuan]

Quick answer: Can you try ./build_release_binaries.sh instead of ./build_and_test.sh in your steps above? Then it should work. The issue you’re encountering has nothing to do with Ubuntu 18.

---

More details:

Earlier today, @akolesnikov and I were just wondering why our internal tests didn’t capture this. We have daily tests that run scripts like this: https://github.com/google/deepvariant/blob/r0.8/scripts/run_wes_case_study_binaries.sh

After checking what that script was doing, I found out that if you run scripts/run_wes_case_study_binaries.sh, it’ll work on both Ubuntu 16 and 18.

We’ll fix build_and_test.sh in the next release. Thanks for reporting! If using build_release_binaries.sh still doesn’t work for you, feel free to reopen.

[pichuan]

Hi @kokyriakidis , in this thread (when it was still r0.8), running ././build-prereq.sh && ./build_release_binaries.sh fixed the original user’s question. In the latest release (r0.9), it should work even if you run build_and_test.sh instead of build_release_binaries.sh.

The main trick was this fix_zip_rule that makes sure the symbolic links are correct: https://github.com/google/deepvariant/blob/97cd861800ccb43d750f392b518e99d514adddd8/build_release_binaries.sh#L41

In the issue that you reported in bcbio/bcbio-nextgen#3048, there is one more layer because you’re using bioconda.

I’m surprised that your issue was actually with v0.9.0. Given that we actually fixed build_and_test.sh in the latest version.

@kokyriakidis Two questions for you:

1. With bioconda, were you able to run with v0.8.0 like @chapmanb suggested?
2. Did you actually try building DeepVariant binaries on your own? If you tried, did that work for you or did you get the same error?