ValueError: The condensed distance matrix must contain only finite values.

Summary:

Dendrogram step failed with an exception, perhaps due to an infinity from a divide by zero?

Description:

Stack trace below.

Reproducible Steps:

I’m not yet sure if this is reproducible, recently I have been re-running average_nucleotide_identity.py on the same input folder (after first unzipping the nucmer tar-ball), while tweaking the input to add a species of interest.

Current Output:

The current output. Knowing what is the current behavior is useful.

$ average_nucleotide_identity.py -i interesting_serratia -o interesting_serratia_cmp -l interesting_serratia_cmp.log -g --labels interesting_serratia/labels.txt --classes interesting_serratia/classes.txt --scheduler SGE --jobprefix "Serratia" --noclobber --force --skip_nucmer
WARNING: NOCLOBBER: not actually deleting directory
WARNING: Skipping NUCmer run (as instructed)!
WARNING: Total alignment length reported in interesting_serratia_cmp/nucmer_output/GCF_000988045.1_GFC_24_genomic_vs_GCA_900016775.1_STsPazieg1.0_genomic.delta is zero!
...
WARNING: Total alignment length reported in interesting_serratia_cmp/nucmer_output/GCF_001514455.1_ASM151445v1_genomic_vs_GCA_900016775.1_STsPazieg1.0_genomic.delta is zero!/mnt/shared/users/pc40583/lib/python3.5/site-packages/scipy/cluster/hierarchy.py:667: RuntimeWarning: invalid value encountered in greater_equal
  if np.all(y >= 0) and np.allclose(y, y.T):
Traceback (most recent call last):
  File "/home/pc40583/bin/average_nucleotide_identity.py", line 4, in <module>
    __import__('pkg_resources').run_script('pyani==0.2.3.dev0', 'average_nucleotide_identity.py')
  File "/home/pc40583/lib/python3.5/site-packages/pkg_resources/__init__.py", line 738, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/pc40583/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1506, in run_script
    exec(script_code, namespace, namespace)
  File "/mnt/shared/users/pc40583/lib/python3.5/site-packages/pyani-0.2.3.dev0-py3.5.egg/EGG-INFO/scripts/average_nucleotide_identity.py", line 811, in <module>
  File "/mnt/shared/users/pc40583/lib/python3.5/site-packages/pyani-0.2.3.dev0-py3.5.egg/EGG-INFO/scripts/average_nucleotide_identity.py", line 640, in draw
  File "/mnt/shared/users/pc40583/lib/python3.5/site-packages/pyani-0.2.3.dev0-py3.5.egg/pyani/pyani_graphics.py", line 322, in heatmap_mpl
  File "/mnt/shared/users/pc40583/lib/python3.5/site-packages/pyani-0.2.3.dev0-py3.5.egg/pyani/pyani_graphics.py", line 200, in add_mpl_dendrogram
  File "/mnt/shared/users/pc40583/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 676, in linkage
    raise ValueError("The condensed distance matrix must contain only finite values.")
ValueError: The condensed distance matrix must contain only finite values.

Expected Output:

I would expect the failure to be caught and a nicer error message to be shown.

Once we understand where the non-finite value comes from, perhaps NaN might work instead…

pyani Version:

$ python3 -c "import pyani; print(pyani.__version__)"
0.2.3.dev

Specifically running from my recently rebased resume branch, see #76

Also relevant,

$ python3 -c "import scipy; print(scipy.__version__)"
0.19.0

Python Version:

$ python3 --version
Python 3.6.0

Operating System:

Linux, 64-bit.

[selahlynch]

I’ve discovered (with help) that this error shows up when an input dataset has columns that contain no information, in other words, they contain only one unique value, such as a column that is all 0’s.

If you are able to access a relevant dataframe, you can check using this code: df.columns[df.nunique() == 1]

[widdowquinn]

Thanks, Selah.

In the next version/release (currently on the development branch) we are moving to an SQLite backend to store results, and we should be able to catch/report this sort of thing earlier, and maybe prevent this error from being shown.

Cheers,

L.