UserWarning: WARNING: spectral initialisation failed! The eigenvector solver failed. This is likely due to too small an eigengap. Consider adding some noise or jitter to your data. Falling back to random initialisation! warn(

While creating umap embeddings for HDBSCAN clustering, I am getting this user warning, `UserWarning: WARNING: spectral initialisation failed! The eigenvector solver failed. This is likely due to too small an eigengap. Consider adding some noise or jitter to your data.

Falling back to random initialisation! warn(` The clusters that are created from these embeddings have multiple duplicate clusters. Why this could be happning, need clarification.

[lmcinnes]

UMAP generates a graph with weighted edges where the edge weights relate to relative distances of neighboring points. You can see the UMAP documentation for more detail on this process. The initialization that is failing is an attempt to find eigenvectors of the (symmetric) Laplacian of that graph. The usual technique for this is to use a power method approach as used by scipy. Scipy, in turn is relying on ARPACK for this. In practical terms all UMAP is seeing is that ARPACK returns and error when attempting to calculate the eigenvectors of the Laplacian. Usually this is related to poor convergence due to a very small gap between eigenvalues so the power method fails to separate out eigenvectors easily. That may or may not be the actual cause in your case – you would have to interrogate the specifics of the ArpackError to be sure.

UMAP does save the graph in the graph_ attribute, so you can actually walk through the code of the spectral layout and catch the actual error and see if that provides more information if you wish.

[lmcinnes]

I think the unique=True option is the best approach; it will find duplicates, remove them for the purpose of learning the embedding, and then place them down exactly as duplicates of their corresponding point in the embedding. So you can run UMAP with duplicates, but not break things – that was the intended use case.

Of course if you want the fact that you have a lot of duplicates to matter for the learned embedding the other simple approach is to simply add a small amount of noise to the whole dataset (such that the scale of the noise is smaller than most variation among (non-duplicated) samples).