RuntimeError: Unable to create link (name already exists) in Image classification

Describe the bug I am trying to create a classification model. When I try to experiment, I get the followong error: RuntimeError: Unable to create link (name already exists)

To Reproduce Steps to reproduce the behavior:

1. I run this command ‘!sudo ludwig experiment --data_csv “/content/drive/My Drive/Colab Notebooks/train.csv” --model_definition_file “/content/drive/My Drive/Colab Notebooks/model_definition.yaml”’
2. the definition_file:

` input_features: - name: Directories type: image encoder: stacked_cnn in_memory: false

output_features: - name: Car_Class type: category

training: batch_size: 8 `

1. Example of train.csv Directories,Car_Class /content/drive/My Drive/Colab Notebooks/car_data/train/Acura Integra Type R 2001/00198.jpg,Acura Integra Type R 2001
2. See error: ----------------------------------------------- `/usr/local/lib/python3.6/dist-packages/ludwig/utils/visualization_utils.py:28: UserWarning: matplotlib.pyplot as already been imported, this call will have no effect. mpl.use(‘TkAgg’) # make matplotlib run on mac os /usr/local/lib/python3.6/dist-packages/matplotlib/init.py:886: MatplotlibDeprecationWarning: examples.directory is deprecated; in the future, examples will be found relative to the ‘datapath’ directory. “found relative to the ‘datapath’ directory.”.format(key))

---

| |_ _ | | __ ()_ _ | | || / \ V V / / _ | ||_,_,|_/_/|_, | |__/ ludwig v0.1.0 - Experiment

Experiment name: experiment Model name: run Output path: results/experiment_run_0

ludwig_version: ‘0.1.0’ command: ('/usr/local/bin/ludwig experiment --data_csv /content/drive/My Drive/Colab ’ 'Notebooks/train.csv --model_definition_file /content/drive/My Drive/Colab ’ ‘Notebooks/model_definition.yaml’) dataset_type: ‘/content/drive/My Drive/Colab Notebooks/train.csv’ model_definition: { ‘combiner’: {‘type’: ‘concat’}, ‘input_features’: [ { ‘encoder’: ‘stacked_cnn’, ‘in_memory’: False, ‘name’: ‘Directories’, ‘should_resize’: False, ‘tied_weights’: None, ‘type’: ‘image’}], ‘output_features’: [ { ‘dependencies’: [], ‘loss’: { ‘class_distance_temperature’: 0, ‘class_weights’: 1, ‘confidence_penalty’: 0, ‘distortion’: 1, ‘labels_smoothing’: 0, ‘negative_samples’: 0, ‘robust_lambda’: 0, ‘sampler’: None, ‘type’: ‘softmax_cross_entropy’, ‘unique’: False, ‘weight’: 1}, ‘name’: ‘Car_Class’, ‘reduce_dependencies’: ‘sum’, ‘reduce_input’: ‘sum’, ‘top_k’: 3, ‘type’: ‘category’}], ‘preprocessing’: { ‘bag’: { ‘fill_value’: ‘’, ‘format’: ‘space’, ‘lowercase’: 10000, ‘missing_value_strategy’: ‘fill_with_const’, ‘most_common’: False}, ‘binary’: { ‘fill_value’: 0, ‘missing_value_strategy’: ‘fill_with_const’}, ‘category’: { ‘fill_value’: ‘<UNK>’, ‘lowercase’: False, ‘missing_value_strategy’: ‘fill_with_const’, ‘most_common’: 10000}, ‘force_split’: False, ‘image’: {‘missing_value_strategy’: ‘backfill’}, ‘numerical’: { ‘fill_value’: 0, ‘missing_value_strategy’: ‘fill_with_const’}, ‘sequence’: { ‘fill_value’: ‘’, ‘format’: ‘space’, ‘lowercase’: False, ‘missing_value_strategy’: ‘fill_with_const’, ‘most_common’: 20000, ‘padding’: ‘right’, ‘padding_symbol’: ‘<PAD>’, ‘sequence_length_limit’: 256, ‘unknown_symbol’: ‘<UNK>’}, ‘set’: { ‘fill_value’: ‘’, ‘format’: ‘space’, ‘lowercase’: False, ‘missing_value_strategy’: ‘fill_with_const’, ‘most_common’: 10000}, ‘split_probabilities’: (0.7, 0.1, 0.2), ‘stratify’: None, ‘text’: { ‘char_format’: ‘characters’, ‘char_most_common’: 70, ‘char_sequence_length_limit’: 1024, ‘fill_value’: ‘’, ‘lowercase’: True, ‘missing_value_strategy’: ‘fill_with_const’, ‘padding’: ‘right’, ‘padding_symbol’: ‘<PAD>’, ‘unknown_symbol’: ‘<UNK>’, ‘word_format’: ‘space_punct’, ‘word_most_common’: 20000, ‘word_sequence_length_limit’: 256}, ‘timeseries’: { ‘fill_value’: ‘’, ‘format’: ‘space’, ‘missing_value_strategy’: ‘fill_with_const’, ‘padding’: ‘right’, ‘padding_value’: 0, ‘timeseries_length_limit’: 256}}, ‘training’: { ‘batch_size’: 8, ‘bucketing_field’: None, ‘decay’: False, ‘decay_rate’: 0.96, ‘decay_steps’: 10000, ‘dropout_rate’: 0.0, ‘early_stop’: 3, ‘epochs’: 200, ‘gradient_clipping’: None, ‘increase_batch_size_on_plateau’: 0, ‘increase_batch_size_on_plateau_max’: 512, ‘increase_batch_size_on_plateau_patience’: 5, ‘increase_batch_size_on_plateau_rate’: 2, ‘learning_rate’: 0.001, ‘learning_rate_warmup_epochs’: 5, ‘optimizer’: { ‘beta1’: 0.9, ‘beta2’: 0.999, ‘epsilon’: 1e-08, ‘type’: ‘adam’}, ‘reduce_learning_rate_on_plateau’: 0, ‘reduce_learning_rate_on_plateau_patience’: 5, ‘reduce_learning_rate_on_plateau_rate’: 0.5, ‘regularization_lambda’: 0, ‘regularizer’: ‘l2’, ‘staircase’: False, ‘validation_field’: ‘combined’, ‘validation_measure’: ‘loss’}}

Using full raw csv, no hdf5 and json file with the same name have been found Building dataset (it may take a while) Traceback (most recent call last): File “/usr/local/bin/ludwig”, line 10, in <module> sys.exit(main()) File “/usr/local/lib/python3.6/dist-packages/ludwig/cli.py”, line 86, in main CLI() File “/usr/local/lib/python3.6/dist-packages/ludwig/cli.py”, line 64, in init getattr(self, args.command)() File “/usr/local/lib/python3.6/dist-packages/ludwig/cli.py”, line 67, in experiment experiment.cli(sys.argv[2:]) File “/usr/local/lib/python3.6/dist-packages/ludwig/experiment.py”, line 548, in cli experiment(**vars(args)) File “/usr/local/lib/python3.6/dist-packages/ludwig/experiment.py”, line 234, in experiment random_seed=random_seed File “/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py”, line 457, in preprocess_for_training random_seed=random_seed File “/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py”, line 62, in build_dataset **kwargs File “/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py”, line 90, in build_dataset_df global_preprocessing_parameters File “/usr/local/lib/python3.6/dist-packages/ludwig/data/preprocessing.py”, line 165, in build_data preprocessing_parameters File “/usr/local/lib/python3.6/dist-packages/ludwig/features/image_feature.py”, line 111, in add_feature_data dtype=np.uint8 File “/usr/local/lib/python3.6/dist-packages/h5py/_hl/group.py”, line 119, in create_dataset self[name] = dset File “/usr/local/lib/python3.6/dist-packages/h5py/_hl/group.py”, line 287, in setitem h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl) File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper File “h5py/h5o.pyx”, line 202, in h5py.h5o.link RuntimeError: Unable to create link (name already exists)` --------------------------------------------------------------------

Environment: absl-py==0.7.0 alabaster==0.7.12 albumentations==0.1.12 altair==2.4.1 astor==0.7.1 astropy==3.0.5 atari-py==0.1.7 atomicwrites==1.3.0 attrs==19.1.0 audioread==2.1.6 autograd==1.2 Babel==2.6.0 backcall==0.1.0 backports.tempfile==1.0 backports.weakref==1.0.post1 beautifulsoup4==4.6.3 bleach==3.1.0 bokeh==1.0.4 boto==2.49.0 boto3==1.9.106 botocore==1.12.106 Bottleneck==1.2.1 bs4==0.0.1 bz2file==0.98 cachetools==3.1.0 certifi==2018.11.29 cffi==1.12.2 chainer==5.0.0 chardet==3.0.4 Click==7.0 cloudpickle==0.6.1 cmake==3.12.0 colorlover==0.3.0 community==1.0.0b1 contextlib2==0.5.5 convertdate==2.1.3 coverage==3.7.1 coveralls==0.5 crcmod==1.7 cufflinks==0.14.6 cupy-cuda100==5.2.0 cvxopt==1.2.3 cvxpy==1.0.15 cycler==0.10.0 cymem==2.0.2 Cython==0.29.6 cytoolz==0.9.0.1 daft==0.0.4 dask==0.20.2 dataclasses==0.6 datascience==0.10.6 decorator==4.3.2 defusedxml==0.5.0 dill==0.2.9 distributed==1.25.3 Django==2.1.7 dlib==19.16.0 dm-sonnet==1.23 docopt==0.6.2 docutils==0.14 dopamine-rl==1.0.5 easydict==1.9 ecos==2.0.7.post1 editdistance==0.5.2 en-core-web-sm==2.0.0 entrypoints==0.3 enum34==1.1.6 ephem==3.7.6.0 et-xmlfile==1.0.1 fa2==0.3.5 fancyimpute==0.4.2 fastai==1.0.46 fastcache==1.0.2 fastdtw==0.3.2 fastprogress==0.1.20 fastrlock==0.4 fbprophet==0.4.post2 featuretools==0.4.1 filelock==3.0.10 fix-yahoo-finance==0.0.22 Flask==1.0.2 folium==0.2.1 future==0.16.0 gast==0.2.2 GDAL==2.2.2 gdown==3.6.4 gensim==3.6.0 geographiclib==1.49 geopy==1.17.0 gevent==1.4.0 gin-config==0.1.4 glob2==0.6 google==2.0.1 google-api-core==1.8.0 google-api-python-client==1.6.7 google-auth==1.4.2 google-auth-httplib2==0.0.3 google-auth-oauthlib==0.2.0 google-cloud-bigquery==1.8.1 google-cloud-core==0.29.1 google-cloud-language==1.0.2 google-cloud-storage==1.13.2 google-cloud-translate==1.3.3 google-colab==1.0.0 google-resumable-media==0.3.2 googleapis-common-protos==1.5.8 googledrivedownloader==0.3 graph-nets==1.0.3 graphviz==0.10.1 greenlet==0.4.15 grpcio==1.15.0 gspread==3.0.1 gspread-dataframe==3.0.2 gunicorn==19.9.0 gym==0.10.11 h5py==2.8.0 HeapDict==1.0.0 holidays==0.9.9 html5lib==1.0.1 httpimport==0.5.16 httplib2==0.11.3 humanize==0.5.1 hyperopt==0.1.2 ideep4py==2.0.0.post3 idna==2.6 image==1.5.27 imageio==2.4.1 imagesize==1.1.0 imbalanced-learn==0.4.3 imblearn==0.0 imgaug==0.2.8 imutils==0.5.2 inflect==2.1.0 intel-openmp==2019.0 intervaltree==2.1.0 ipykernel==4.6.1 ipython==5.5.0 ipython-genutils==0.2.0 ipython-sql==0.3.9 ipywidgets==7.4.2 itsdangerous==1.1.0 jdcal==1.4 jedi==0.13.3 jieba==0.39 Jinja2==2.10 jmespath==0.9.4 joblib==0.12.5 jpeg4py==0.1.4 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.4 jupyter-console==6.0.0 jupyter-core==4.4.0 kaggle==1.5.3 kapre==0.1.3.1 Keras==2.2.4 Keras-Applications==1.0.7 Keras-Preprocessing==1.0.9 keras-vis==0.4.1 kiwisolver==1.0.1 knnimpute==0.1.0 librosa==0.6.3 lightgbm==2.2.3 llvmlite==0.27.1 lmdb==0.94 lucid==0.3.8 ludwig==0.1.0 lunardate==0.2.0 lxml==4.2.6 magenta==0.3.19 Markdown==3.0.1 MarkupSafe==1.1.1 matplotlib==3.0.0 matplotlib-venn==0.11.5 mesh-tensorflow==0.0.5 mido==1.2.6 mir-eval==0.5 missingno==0.4.1 mistune==0.8.4 mkl==2019.0 mlxtend==0.14.0 mock==2.0.0 more-itertools==6.0.0 moviepy==0.2.3.5 mpi4py==3.0.1 mpmath==1.1.0 msgpack==0.5.6 msgpack-numpy==0.4.3.2 multiprocess==0.70.7 multitasking==0.0.7 murmurhash==1.0.2 music21==5.5.0 natsort==5.5.0 nbconvert==5.4.1 nbformat==4.4.0 networkx==2.2 nibabel==2.3.3 nltk==3.2.5 nose==1.3.7 notebook==5.2.2 np-utils==0.5.10.0 numba==0.40.1 numexpr==2.6.9 numpy==1.14.6 nvidia-ml-py3==7.352.0 oauth2client==4.1.3 oauthlib==3.0.1 okgrade==0.4.3 olefile==0.46 opencv-contrib-python==3.4.3.18 opencv-python==3.4.5.20 openpyxl==2.5.9 osqp==0.5.0 packaging==19.0 pandas==0.22.0 pandas-datareader==0.7.0 pandas-gbq==0.4.1 pandas-profiling==1.4.1 pandocfilters==1.4.2 parso==0.3.4 pathlib==1.0.1 patsy==0.5.1 pbr==5.1.3 pexpect==4.6.0 pickleshare==0.7.5 Pillow==4.1.1 pip-tools==3.4.0 plac==0.9.6 plotly==3.6.1 pluggy==0.7.1 portpicker==1.2.0 prefetch-generator==1.0.1 preshed==2.0.1 pretty-midi==0.2.8 prettytable==0.7.2 progressbar2==3.38.0 prometheus-client==0.6.0 promise==2.2.1 prompt-toolkit==1.0.15 protobuf==3.6.1 psutil==5.4.8 psycopg2==2.7.6.1 ptyprocess==0.6.0 py==1.8.0 pyasn1==0.4.5 pyasn1-modules==0.2.4 pycocotools==2.0.0 pycparser==2.19 pydot==1.3.0 pydot-ng==2.0.0 pydotplus==2.0.2 pyemd==0.5.1 pyglet==1.3.2 Pygments==2.1.3 pygobject==3.26.1 pymc3==3.6 pymongo==3.7.2 pymystem3==0.2.0 PyOpenGL==3.1.0 pyparsing==2.3.1 pyrsistent==0.14.11 pysndfile==1.3.2 PySocks==1.6.8 pystache==0.5.4 pystan==2.18.1.0 pytest==3.6.4 python-apt==1.6.3+ubuntu1 python-chess==0.23.11 python-dateutil==2.5.3 python-louvain==0.13 python-rtmidi==1.2.1 python-slugify==3.0.0 python-utils==2.3.0 pytz==2018.9 PyWavelets==1.0.2 PyYAML==3.13 pyzmq==17.0.0 qtconsole==4.4.3 regex==2018.1.10 requests==2.18.4 requests-oauthlib==1.2.0 resampy==0.2.1 retrying==1.3.3 rpy2==2.9.5 rsa==4.0 s3fs==0.2.0 s3transfer==0.2.0 scikit-image==0.13.1 scikit-learn==0.20.3 scipy==1.1.0 screen-resolution-extra==0.0.0 scs==2.0.2 seaborn==0.7.1 Send2Trash==1.5.0 setuptools-git==1.2 Shapely==1.6.4.post2 simplegeneric==0.8.1 six==1.11.0 sklearn==0.0 smart-open==1.8.0 snowballstemmer==1.2.1 sortedcontainers==2.1.0 spacy==2.0.18 Sphinx==1.8.4 sphinxcontrib-websupport==1.1.0 SQLAlchemy==1.2.18 sqlparse==0.2.4 stable-baselines==2.2.1 statsmodels==0.8.0 sympy==1.1.1 tables==3.4.4 tabulate==0.8.3 tblib==1.3.2 tensor2tensor==1.11.0 tensorboard==1.13.0 tensorboardcolab==0.0.22 tensorflow==1.13.1 tensorflow-estimator==1.13.0 tensorflow-hub==0.3.0 tensorflow-metadata==0.12.1 tensorflow-probability==0.6.0 termcolor==1.1.0 terminado==0.8.1 testpath==0.4.2 text-unidecode==1.2 textblob==0.15.3 textgenrnn==1.4.1 tfds-nightly==1.0.1.dev201903040105 tflearn==0.3.2 Theano==1.0.4 thinc==6.12.1 toolz==0.9.0 torch==1.0.1.post2 torchsummary==1.5.1 torchtext==0.3.1 torchvision==0.2.2.post3 tornado==4.5.3 tqdm==4.28.1 traitlets==4.3.2 tweepy==3.6.0 typing==3.6.6 tzlocal==1.5.1 ujson==1.35 umap-learn==0.3.7 uritemplate==3.0.0 urllib3==1.22 vega-datasets==0.7.0 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.14.1 widgetsnbextension==3.4.2 wordcloud==1.5.0 wrapt==1.10.11 xarray==0.11.3 xgboost==0.7.post4 xkit==0.0.0 xlrd==1.1.0 xlwt==1.3.0 yellowbrick==0.9.1 zict==0.1.3 zmq==0.0.0

[w4nderlust]

It’s still 0.1.0 yes. If you resize the images, but the tensors mismatch, the only explanation that I can think of is that the first time you run it you did the preprocessing and saved into hdf5 without resizing. So please delete the hdf5 and json file from the directory where the data lives and start again an experiment.

[w4nderlust]

@yl642 and @Belginwls you need to provide some explanation for the error, just posting those 4 lines does not make it possible for us to help you, we are developers not magicians 😃 If you provide more context and replicable setup i can look into it, maybe you can open a different issue for that once you ave a reproducible example of the problem.