Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Issue running parametric umap on large data - proto limit

See original GitHub issue

I’m trying to run a basic parametric UMAP embedding on my dataset. Each image is (193, 229, 193) so flattened out is 8,530,021 dimensions.

This is the error when trying to run embedding = embedder.fit_transform(train_images).

raise ValueError(

    ValueError: Tried to convert 'params' to a tensor and failed. Error: Cannot create a tensor proto whose content is larger than 2GB.

It seems like this is just a limitation of tensorflow so I am not sure what can be done about this on your end. But I was wondering if anyone has faced something similar with large datasets and if there was a workaround for this issue?

Issue Analytics

State:
Created 3 years ago
Comments:8

Top GitHub Comments

1reaction

timsainbcommented, Mar 11, 2021

It looks like the issue here is the tensorflow dataset / iterator being created from a numpy array, which has an upper limit of 2GB. We could try a different iterator, which would move the upper limit to the amount of data that fits in RAM. [1] https://stackoverflow.com/a/53382823/200663 [2] https://stackoverflow.com/a/55126482/200663

1reaction

timsainbcommented, Mar 10, 2021

Have you tried decreasing batch_size?

Top Results From Across the Web

Tried Parametric UMAP but its performance does not seem to ...

Problem : I am trying to perform text clustering using Sentence Transformers embedding of 748 dimensions. Method: I have supervised data of around...

Frequently Asked Questions — umap 0.5 documentation

If your dataset is not especially large but you have found that UMAP runs out of memory when operating on it consider using...

A review of UMAP in population genetics - Nature

UMAP allows for specification of a minimum distance between nearest neighbours in low-dimensional space: higher values are useful for ...

Use of “default” parameter settings when analyzing single cell ...

The pipeline for performing unbiased cell clustering within the Seurat pipeline is: (1) filter the dataset based on minimum/maximum cut-offs for genes/cell, ...

Processing single-cell RNA-seq datasets using SingCellaR

This protocol describes a method for analyzing single-cell RNA sequencing (scRNA-seq) datasets using an R package called SingCellaR. In addition ...