KeyError: 'test_dir'

Hi, I’ following the documentation and encountered problems: In section “Evaluate and visualize 6D pose estimation of AAE with ground truth bounding boxes” after I run the following command:

ae_eval exp_group/my_autoencoder evaluation --eval_cfg eval_group/eval_my_autoencoder.cfg

(I didn’t modify the configuration file template, since estimate_bbs is already set to False) I get following errors :

Processing: /localhome/demo/autoencoder_6d_pose_estimation/AugmentedAutoencoder_ws/cfg_eval/eval_results/hodan-iros15_tless_primesense
test_primesense
Loading object models...

Done.
{1: [5, 7, 9, 17, 18, 20], 2: [1, 12, 20, 9], 3: [9, 12, 20, 7], 4: [9, 18, 20, 5, 17], 5: [11, 2, 3, 4], 6: [2, 6], 7: [17, 2, 12, 18, 6], 8: [11, 3, 4], 9: [17, 18, 11, 12, 5], 10: [16, 11, 5], 11: [16, 3, 6], 12: [16, 3, 6], 13: [16, 19, 7], 14: [16, 19, 7], 15: [16, 19, 7], 16: [16, 19, 7], 17: [16, 19, 7], 18: [19, 3, 7], 19: [8, 10, 13, 14], 20: [8, 10, 13, 14], 21: [8, 10, 13], 22: [8, 10, 14], 23: [8, 10, 13, 14], 24: [8, 10, 19, 14], 25: [1, 15], 26: [4, 15], 27: [5, 15], 28: [4, 13, 15], 29: [1, 15], 30: [1, 19, 15]}
test_primesense
<backports.configparser.ConfigParser object at 0x7f631558cf50>
128 128 3
[[8, 8], [16, 16], [32, 32], [64, 64]]
(?, 128, 128, 3)
(?, 128, 128, 3)
2019-02-25 17:57:11.296768: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-25 17:57:11.373947: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-25 17:57:11.374378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.506
pciBusID: 0000:01:00.0
totalMemory: 3.94GiB freeMemory: 2.89GiB
2019-02-25 17:57:11.374393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-25 17:57:11.542753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-25 17:57:11.542794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-02-25 17:57:11.542799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-02-25 17:57:11.542967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2019 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
11
scene_id: 11
eval_args: <backports.configparser.ConfigParser object at 0x7f631558cf50>
test_primesense
dataset_name: tless
cam_type: primesense
test_primesense

Traceback (most recent call last):
  File "/localhome/demo/autoencoder_6d_pose_estimation/venv/bin/ae_eval", line 11, in <module>
    load_entry_point('auto-pose==0.9', 'console_scripts', 'ae_eval')()
  File "/localhome/demo/autoencoder_6d_pose_estimation/venv/local/lib/python2.7/site-packages/auto_pose/eval/ae_eval.py", line 114, in main
    test_imgs = eval_utils.load_scenes(scene_id, eval_args)
  File "/localhome/demo/autoencoder_6d_pose_estimation/venv/local/lib/python2.7/site-packages/auto_pose/eval/eval_utils.py", line 152, in load_scenes
    noof_imgs = noof_scene_views(scene_id, eval_args)
  File "/localhome/demo/autoencoder_6d_pose_estimation/venv/local/lib/python2.7/site-packages/auto_pose/eval/eval_utils.py", line 138, in noof_scene_views
    noof_imgs = len(os.listdir(os.path.join(p['base_path'], p['test_dir'], '{:02d}', 'rgb').format(scene_id)))
KeyError: 'test_dir'

I set up my sixd_toolkit as follows:

pip install -r requirements.txt
in dataset_params.py:
common_base_path = '/localhome/demo/autoencoder_6d_pose_estimation/t-less/t-less_v2/t-less_v2/'
tless_tk_path = '/localhome/demo/autoencoder_6d_pose_estimation/t-less_toolkit/'

Thanks a lot if someone can help me out…

System Info

• GPU 1050 4G
• Python version:2.7.12
• PyOpenGL version: 3.1.3b2

by the way it seems that there’s a little mistake in Create the evaluation config file:

should be:  mkdir $AE_WORKSPACE_PATH/cfg_eval/eval_group,  instead of ../eval_cfg/..
cp $AE_WORKSPACE_PATH/cfg_eval/eval_template.cfg $AE_WORKSPACE_PATH/eval_cfg/eval_group/eval_my_autoencoder.cfg
gedit $AE_WORKSPACE_PATH/cfg_eval/eval_group/eval_my_autoencoder.cfg ---- instead of ../cfg/..

[MartinSmeyer]

Yes, these are the ground truth visibility statistics of t-less. You are right they are not part of the original dataset. Download them here

http://ptak.felk.cvut.cz/6DB/public/datasets/t-less/

And place the files where they are currently not found.

[MartinSmeyer]

I have updated the repo with instructions to reproduce the BOP19 results.

https://github.com/DLR-RM/AugmentedAutoencoder#reproducing-and-visualizing-bop-challenge-results

Since BOP is the standard now, it should be used for comparisons in new works.