Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Resources required?

See original GitHub issue

Hello, Exciting project. Could you please share the minimum resource requirements to train this model? I am getting memory errors training 500,000 256x256 images on 4 40GB A100 GPU’s. Also my CPU memory is 128GB.

I tensorflow/stream_executor/stream.cc:1990] [stream=0x5570d9fd4600,impl=0x5570d9fd2050] did not wait for [stream=0x5570d9fd40a0,impl=0x5570d9fd2940] 2021-05-26 01:09:18.033272: I tensorflow/stream_executor/stream.cc:4925] [stream=0x5570d9fd4600,impl=0x5570d9fd2050] did not memcpy device-to-host; source: 0x2ac3e5d41a00 2021-05-26 01:09:18.033313: F tensorflow/core/common_runtime/gpu/gpu_util.cc:293] GPU->CPU Memcpy failed /hpc/users/marxg01/.lsbatch/1622001386.34151325.shell: line 23: 389547 Aborted (core dumped)

Issue Analytics

State:
Created 2 years ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

gabemarxcommented, May 28, 2021

The problem was fixed by deprecating my Cuda version from 11.1 to 7.0.28

Appreciate the help!

0reactions

LynnHocommented, May 27, 2021

Ok, I will check the code. If possible, by the way, I suggest using the dataset of this code to check whether the error only happens with your own data. And maybe try CUDA_VISIBLE_DEVICES=0,1 to use fewer GPUs, using all GPUs of a device may cause an error (I don’t know why).

Top Results From Across the Web

Define resource requirements | Microsoft Learn

Resource requirements are defined by the Project manager to establish the resources needed to execute the work on the project.

Determine Required Resources - The Project Management ...

The process requires both an overall assessment of the amount of each type of resource and specific identification of skills, responsibility, and details....

3 Types of Essential Resources For Your Project - PMTips

Project resources are defined as the people, capital, and material or supplies needed for successful management and completion of a project.

Resource (project management) - Wikipedia

In project management, resources are required to carry out the project tasks. These can be people, equipment, facilities, funding, or anything else capable ......

Your ultimate guide to project management resources

If you remember from 2 sections ago, project resources can be divided into 7 categories: people, information, materials, tools, energy, money, ...