OOM error when use for loop

I have tried to use the suggestion codel as follows to calculate the distance between two faces.

model_name = "Facenet"
model = DeepFace.build_model(model_name)
DeepFace.verify("img1.jpg", "img2.jpg", model_name = model_name, model = model)

But I got the error after 1117/206040 iterations:

ResourceExhaustedError: OOM when allocating tensor with shape[1,613,613,10] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node p_re_lu_42385/Relu_1-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. [Op:__inference_keras_scratch_graph_3394553]

Function call stack: keras_scratch_graph

I have four 2080 Ti on my server, but it looks like it just uses the first GPU with full memory used. So I am wondering whether the memory of GPU causes this problem? Any suggestions will be appreciated.

The code as follows:

model_name = "ArcFace"
model = DeepFace.build_model(model_name)

pairs_distance_lists = []

for j in range(len(pairs_name_dict)):
    
    print('{now}/{total}'.format(now = j, total = len(pairs_name_dict)))
    
    path1 = pairs_dict[j][0] # the path of frist image  e.g './Morphed/001_03_R_morphed_0.41_002_03_R.png'
    path2 = pairs_dict[j][1] # the path of second image e.g '../datasets/AMSL/londondb_genuine_neutral_passport-scale_15kb/001_03_q30.jpg'
    
    result = DeepFace.verify(path1, path2, model_name=model_name, model=model,\
                              distance_metric = "cosine", detector_backend = 'mtcnn')

    # extract distance from result
    a1 = pairs_name_dict[j][0]
    a2 = pairs_name_dict[j][1]
    a3 = result['distance']
    pairs_distance = [a1, a2, a3]
    print(pairs_distance)
    pairs_distance_lists.append(pairs_distance)

Thanks

[serengil]

That’s memory based error. But you do everything based on the best practice approaches. Unfortunately, I do not have a gpu and test this case.

Please read this post: https://sefiks.com/2019/03/20/tips-and-tricks-for-gpu-and-multiprocessing-in-tensorflow/

1- Try to set all devices. This should be defined before importing deepface.

import os os.environ[“CUDA_VISIBLE_DEVICES”]=“0,1,2,3”

2- Disable allocating all memory. Let it to allocate how much it needs.

config = tf.ConfigProto() config.gpu_options.allow_growth = True session = tf.Session(config=config) keras.backend.set_session(session)

3- change detector_backend to opencv. that causes same error?

[serengil]

I just published deepface 0.0.60. Many production-driven performance issues are handled in this release. Please update the package and re-try.