Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Can't generate samples from checkpoint file

See original GitHub issue

When I try to run generate.py per the readme, I get this:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
Restoring model from model.ckpt-250
Traceback (most recent call last):
  File "generate.py", line 86, in <module>
    main()
  File "generate.py", line 66, in main
    feed_dict={samples: window})
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Output dimensions must be positive
         [[Node: wavenet/dilated_stack/layer1/conv_filter/BatchToSpace = BatchToSpace[T=DT_FLOAT, block_size=2, _device="/job:localhost/replica:0/task:0/gpu:0"](wavenet/dilated_stack/layer1/conv_filter, wavenet/dilated_stack/layer1/conv_filter/BatchToSpace/crops)]]
Caused by op u'wavenet/dilated_stack/layer1/conv_filter/BatchToSpace', defined at:
  File "generate.py", line 86, in <module>
    main()
  File "generate.py", line 51, in main
    next_sample = net.predict_proba(samples)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 154, in predict_proba
    raw_output = self._create_network(encoded)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 112, in _create_network
    self.dilation_channels)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 51, in _create_dilation_layer
    name="conv_filter")
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 228, in atrous_conv2d
    block_size=rate)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 308, in batch_to_space
    block_size=block_size, name=name)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()```

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Reactions:6
  • Comments:22 (16 by maintainers)

github_iconTop GitHub Comments

6reactions
maxhodakcommented, Sep 16, 2016

Thanks for taking a look at this! I no longer get the above error, but now get:

I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:03:00.0
Total memory: 11.92GiB
Free memory: 11.81GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
Restoring model from model.ckpt-1950
Traceback (most recent call last):
  File "generate.py", line 86, in <module>
    main()
  File "generate.py", line 66, in main
    feed_dict={samples: window})
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 710, in run
    run_metadata_ptr)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 908, in _run
    feed_dict_string, options, run_metadata)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 958, in _do_run
    target_list, options, run_metadata)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 978, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Expected begin[2] == 0 (got 0) and size[2] == 0 (got -2) when input.dim_size(2) == 0
     [[Node: wavenet/dilated_stack/layer2/Slice = Slice[Index=DT_INT32, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](wavenet/dilated_stack/layer2/Reshape_1, wavenet/dilated_stack/layer2/Slice/begin, wavenet/dilated_stack/layer2/Slice/size)]]
     [[Node: wavenet/Reshape_1/_131 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_2304_wavenet/Reshape_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'wavenet/dilated_stack/layer2/Slice', defined at:
  File "generate.py", line 86, in <module>
    main()
  File "generate.py", line 51, in main
    next_sample = net.predict_proba(samples)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 171, in predict_proba
    raw_output = self._create_network(encoded)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 129, in _create_network
    self.dilation_channels)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 75, in _create_dilation_layer
    conv_filter = self._causal_dilated_conv(input_batch, weights_filter, dilation)
  File "/home/ubuntu/jupyter_base/project/tensorflow-wavenet/wavenet.py", line 48, in _causal_dilated_conv
    out = tf.slice(restored, 4 * [0], [-1, -1, tf.shape(restored)[2] - pad_elements, -1])
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 328, in slice
    return gen_array_ops._slice(input_, begin, size, name=name)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2009, in _slice
    name=name)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2317, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/ubuntu/jupyter_base/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1239, in __init__
    self._traceback = _extract_stack()
3reactions
danbricommented, Sep 16, 2016

Trying even after pulling these patches and retraining (although not for long), I get this (OSX, no GPU):

22:43 $ python generate.py --samples 16000 model.ckpt-150
Restoring model from model.ckpt-150
E tensorflow/core/client/tensor_c_api.cc:485] Expected begin[2] == 0 (got 0) and size[2] == 0 (got -2) when input.dim_size(2) == 0
     [[Node: wavenet/dilated_stack/layer2/causal_conv/Slice_6 = Slice[Index=DT_INT32, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](wavenet/dilated_stack/layer2/causal_conv/Reshape_1, wavenet/dilated_stack/layer2/causal_conv/Slice_6/begin, wavenet/dilated_stack/layer2/causal_conv/Slice_6/size)]]
Traceback (most recent call last):
  File "generate.py", line 86, in <module>
    main()
  File "generate.py", line 66, in main
    feed_dict={samples: window})
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Expected begin[2] == 0 (got 0) and size[2] == 0 (got -2) when input.dim_size(2) == 0
     [[Node: wavenet/dilated_stack/layer2/causal_conv/Slice_6 = Slice[Index=DT_INT32, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](wavenet/dilated_stack/layer2/causal_conv/Reshape_1, wavenet/dilated_stack/layer2/causal_conv/Slice_6/begin, wavenet/dilated_stack/layer2/causal_conv/Slice_6/size)]]
Caused by op u'wavenet/dilated_stack/layer2/causal_conv/Slice_6', defined at:
  File "generate.py", line 86, in <module>
    main()
  File "generate.py", line 51, in main
    next_sample = net.predict_proba(samples)
  File "/Users/danbri/tensorflow-wavenet/wavenet.py", line 174, in predict_proba
    raw_output = self._create_network(encoded)
  File "/Users/danbri/tensorflow-wavenet/wavenet.py", line 132, in _create_network
    self.dilation_channels)
  File "/Users/danbri/tensorflow-wavenet/wavenet.py", line 78, in _create_dilation_layer
    conv_filter = self._causal_dilated_conv(input_batch, weights_filter, dilation)
  File "/Users/danbri/tensorflow-wavenet/wavenet.py", line 49, in _causal_dilated_conv
    out = tf.slice(restored, 4 * [0], [-1, -1, tf.shape(restored)[2] - pad_elements, -1])
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 388, in slice
    return gen_array_ops._slice(input_, begin, size, name=name)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2001, in _slice
    name=name)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2310, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/danbri/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1232, in __init__
    self._traceback = _extract_stack()
Read more comments on GitHub >

github_iconTop Results From Across the Web

Unable to load weights from pytorch checkpoint file #4336
In my case, there was something problem during moving the files, so pytorch_model.bin file existed but the size was 0 byte. After replacing...
Read more >
checkpoint files not getting created - CosmoCoffee
Infact, I used supernovae data likelihood module as the template. I have 3 extra parameters which are entirely handled in the data likelihood ......
Read more >
Training checkpoints | TensorFlow Core
Use a tf. train. Checkpoint object to manually create a checkpoint, where the objects you want to checkpoint are set as attributes on...
Read more >
Using Gaussian Checkpoint Files
When a calculation is started using information from a checkpoint file, the calculation results will be placed in the exact same checkpoint file,...
Read more >
Using a CHECKPOINT in SSIS packages to restart package ...
Open the file, and you can see the content of this file as follows. Fix the error message and rerun the package.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Add fast wavenet generation.

Next page

Excessive memory consumption