Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

not able to generate waymo_processed_data

See original GitHub issue

@sshaoshuai Thank you for all the continued work you put into this repository!

I am facing issues when I am trying to generate waymo_processed_data following your instructions in Getting Started.

I tried it on a Tesla K80 first and it failed, so I thought it is maybe because the GPU is too old, so I tried it on a Tesla V100 next, but here it fails, too.

The two error logs are listed below. Any idea what could be the issue?

(PCDet) ubuntu /home/ubuntu/efs/repositories/PCDet 
→ python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
    --cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml

Tesla K80 K80.log (too large to put in line)

Tesla V100

/home/ubuntu/anaconda3/envs/PCDet/lib/python3.6/runpy.py:125: RuntimeWarning: 'pcdet.datasets.waymo.waymo_dataset' found in sys.modules after import of package 'pcdet.datasets.waymo', but prior to execution of 'pcdet.datasets.waymo.waymo_dataset'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
/home/ubuntu/efs/repositories/PCDet/pcdet/datasets/waymo/waymo_dataset.py:360: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  dataset_cfg = EasyDict(yaml.load(open(args.cfg_file)))
2020-11-10 16:41:45,781   INFO  Loading Waymo dataset
2020-11-10 16:41:45,800   INFO  Total skipped info 202
2020-11-10 16:41:45,800   INFO  Total samples for Waymo dataset: 0
2020-11-10 16:41:45,800   INFO  Total sampled samples for Waymo dataset: 0
2020-11-10 16:41:46,039   INFO  Loading Waymo dataset
2020-11-10 16:41:46,074   INFO  Total skipped info 798
2020-11-10 16:41:46,074   INFO  Total samples for Waymo dataset: 0
2020-11-10 16:41:46,074   INFO  Total sampled samples for Waymo dataset: 0
2020-11-10 16:41:47.870777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-10 16:41:47.899821: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:47.900781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
2020-11-10 16:41:47.900824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-10 16:41:47.902699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-10 16:41:47.904293: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-10 16:41:47.904664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-10 16:41:47.906729: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-10 16:41:47.908282: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-10 16:41:47.908333: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-10 16:41:47.908423: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:47.909394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:47.910276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-10 16:41:47.912516: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-10 16:41:47.921158: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300080000 Hz
2020-11-10 16:41:47.921714: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb678444140 executing computations on platform Host. Devices:
2020-11-10 16:41:47.921736: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2020-11-10 16:41:47.921935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:47.922864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
2020-11-10 16:41:47.922902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-10 16:41:47.922925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-10 16:41:47.922943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-10 16:41:47.922960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-10 16:41:47.922977: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-10 16:41:47.922994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-10 16:41:47.923008: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-10 16:41:47.923070: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:47.923993: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:47.924856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-10 16:41:50.566097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-10 16:41:50.566155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-11-10 16:41:50.566164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-11-10 16:41:50.566443: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:50.567417: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:50.568333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-10 16:41:50.569241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14638 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2020-11-10 16:41:50.571892: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb56947d210 executing computations on platform CUDA. Devices:
2020-11-10 16:41:50.571913: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2020-11-10 16:41:52.543438: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
2020-11-10 16:41:52.543505: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
2020-11-10 16:41:52.828328: I tensorflow/stream_executor/stream.cc:1868] [stream=0x7fb678d712a0,impl=0x7fb678d6f7d0] did not wait for [stream=0x7fb678d71060,impl=0x7fb678d6f800]
2020-11-10 16:41:52.828395: I tensorflow/stream_executor/stream.cc:4816] [stream=0x7fb678d712a0,impl=0x7fb678d6f7d0] did not memcpy host-to-device; source: 0x7fb121f06b40
2020-11-10 16:41:52.828529: F tensorflow/core/common_runtime/gpu/gpu_util.cc:342] CPU->GPU Memcpy failed

The necessary imports seem to work fine:

(PCDet) ubuntu /home/ubuntu/efs/repositories/PCDet 
→ python
Python 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> import waymo_open_dataset as wod
>>> print(tf.__version__)
2.0.0

On my local machine with a GeForce GTX 1060, I can not yet debug the code, unfortunately, because there I am still in the process of downloading and extracting the dataset.