New error: Resource exhausted

After using CARE for over 6 months all of a sudden I start getting an error when simply running model.predict:

ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[1,32,1024,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node down_level_0_no_0/convolution}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[1,32,1024,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node down_level_0_no_0/convolution}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[concatenate_3/concat/_291]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

I have two GPUs in my laptop (Intel UHD Graphics 620 and NVIDIA GeForce MX130) and for some reason (maybe it always had I never checked) in the task manager and the error above tensorflow uses the Intel one.

I have tried using:

with tf.device('/gpu:1'): 
    a = model.predict(b, axes ='YX')

but I get the same error.

Any advice would be highly appreciated. Strange that it happens now all of a sudden since I am running it on identical size files

[joaomamede]

Just trying to help with the above problem. I don’t have this problem with csbdeep.

[uschmidt83]

Hi @joaomamede, I don’t know why you’re replying to this issue, but try this.