KeyError: "['file_path_mask_eroded_3'] not in index"

When running local pure python with python main.py -- train_evaluate_predict --pipeline_name unet --chunk_size 5000 , the following error occurs, any help?

neptune: Executing in Offline Mode. neptune: Executing in Offline Mode. 2018-05-29 16-16-52 mapping-challenge >>> training neptune: Executing in Offline Mode. 2018-05-29 16-16-55 steps >>> step xy_train adapting inputs 2018-05-29 16-16-55 steps >>> step xy_train fitting and transforming… Traceback (most recent call last): File “main.py”, line 282, in <module> action() File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/click/core.py”, line 722, in call return self.main(*args, **kwargs) File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/click/core.py”, line 697, in main rv = self.invoke(ctx) File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/click/core.py”, line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/click/core.py”, line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/click/core.py”, line 535, in invoke return callback(*args, **kwargs) File “main.py”, line 79, in train _train(pipeline_name, dev_mode) File “main.py”, line 106, in _train pipeline.fit_transform(data) File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/base.py”, line 103, in fit_transform step_inputs[input_step.name] = input_step.fit_transform(data) File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/base.py”, line 103, in fit_transform step_inputs[input_step.name] = input_step.fit_transform(data) File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/base.py”, line 103, in fit_transform step_inputs[input_step.name] = input_step.fit_transform(data) [Previous line repeated 5 more times] File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/base.py”, line 109, in fit_transform step_output_data = self._cached_fit_transform(step_inputs) File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/base.py”, line 120, in _cached_fit_transform step_output_data = self.transformer.fit_transform(**step_inputs) File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/base.py”, line 253, in fit_transform return self.transform(*args, **kwargs) File “/media/rs/3EBAC1C7BAC17BC1/Xavier/crowdAI/open-solution-mapping-challenge/steps/preprocessing/misc.py”, line 17, in transform y = meta[self.y_columns].values File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/pandas/core/frame.py”, line 2133, in getitem return self._getitem_array(key) File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/pandas/core/frame.py”, line 2177, in _getitem_array indexer = self.loc._convert_to_indexer(key, axis=1) File “/home/rs/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/pandas/core/indexing.py”, line 1269, in _convert_to_indexer .format(mask=objarr[mask])) KeyError: “[‘file_path_mask_eroded_3’] not in index”

[jakubczakon]

Well, you can simply use predict_on_dir which takes a directory of images as input:

python main.py predict_on_dir \
--pipeline_name unet_tta_scoring_model \
--chunk_size 1000 \
--dir_path path/to/inference_directory \
--prediction_path path/to/predictions.json

That will get you predicted segmentation masks which you can later plot by using results exploration notebook.

It is not a proper evaluation but it is definitely quicker.

[jakubczakon]

@XYAskWhy @dslwz2008 I will fix the readme today but yes as @XYAskWhy when metadata is created it looks for the folders with target masks and creates the columns based on that information. It may seem over the top at first glance but creating target masks for this problem is very far from trivial. The following ideas are all viable options:

• overlay target masks
• erode masks first and overlay
• erode large masks but dilate small masks and overlay (to increase signal for the small objects)
• drop border masks that are very thin (<2 pixels) and then overlay to decrease false signals from mislabeled edge objects

I hope this helps!