semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache))

Bug Description

Hi, everyone! When I try to use autokeras fit some data and label, I encounter such warnings as: “semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache))”

After some iterations, for example, after prompt “Model 9”, the program is hanging due to insufficient memory.

I don’t think it has connection with my code, here is my code:

from autokeras.image_supervised import ImageClassifier
import numpy as np

if __name__ == '__main__':
    x_train = np.load('x_train.npy')
    y_train = np.load('y_train.npy')
    x_test = np.load('x_test.npy')
    y_test = np.load('y_test.npy')
    x_train = x_train.reshape((-1, 32, 32, 3))
    y_train.reshape((-1, 1))
    x_test = x_test.reshape((-1, 32, 32, 3))
    y_test = y_test.reshape((-1, 1))
    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12*60*60)
    clf.final_fit(x_train, y_train, x_test, y_test, retrain=True)
    pred = clf.evaluate(x_test, y_test)
    print(pred)

The magnitude of my dataset is similar to CIFAR10.

I want to know, how to fix this warning and following memory issue? Does anyone has experience in fixing such things? Thanks in advance.

Reproducing Steps

Steps to reproduce the behavior:

• Step 1: …
• Step 2: …

Expected Behavior

Setup Details

Include the details about the versions of:

• OS type and version: Ubuntu 16.04
• Python: 3.6
• autokeras:
• scikit-learn:
• numpy:
• keras: 2.2.2
• scipy:
• tensorflow: 1.10.0
• pytorch: All requirements are installed by “pip install autokeras” and I run my code in a Docker instance, Some info on my Docker environment: chuanming@Galadriel:~$ nvidia-docker version NVIDIA Docker: 2.0.3 Client: Version: 18.03.1-ce API version: 1.37 Go version: go1.9.5 Git commit: 9ee9f40 Built: Thu Apr 26 07:17:20 2018 OS/Arch: linux/amd64 Experimental: false Orchestrator: swarm

Server: Engine: Version: 18.03.1-ce API version: 1.37 (minimum version 1.12) Go version: go1.9.5 Git commit: 9ee9f40 Built: Thu Apr 26 07:15:30 2018 OS/Arch: linux/amd64 Experimental: false

Additional context

[monikatomar92]

I experienced the same warning. UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache))

[anirban0104]

I am getting this same error while running a ML model in deepfacelab colab, and after the error, the cell stops executing.