Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

training: Cannot convert a symbolic Tensor to a numpy array

See original GitHub issue

I have Calamari 2.2.1 / Python 3.7.4 / TF 2.4.1 / Numpy 1.21.6, and attempt calamari-train in a (Power9-based) HPC environment.

Unfortunately, I get the following exception:

stacktrace 
INFO     2022-04-26 18:48:37,860          tfaip.trainer.trainer: Setting samples per epoch relative to dataset size with a factor of 1.
INFO     2022-04-26 18:48:37,860          tfaip.trainer.trainer: Set steps per epoch to 18
INFO     2022-04-26 18:48:37,874    tfaip.scenario.scenariobase: Building training keras model
INFO     2022-04-26 18:48:37,881    tfaip.scenario.scenariobase: Attempting to set no train scope
INFO     2022-04-26 18:48:37,881    tfaip.scenario.scenariobase: Compiling training model including optimization
INFO     2022-04-26 18:48:37,892    tfaip.scenario.scenariobase: Interference on the input shapes to build the graph.
CRITICAL 2022-04-26 18:48:39,339             tfaip.util.logging: Uncaught exception
Traceback (most recent call last):
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main
    return trainer.train()
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train
    last_logs = super().train(callbacks=callbacks)
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper
    return func(*args, **kwargs)
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train
    training_graph_only=self._training_graph_only,
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training
    self._keras_train_model((real_inputs, wrapped_targets, real_meta))
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__
    input_list)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call
    inputs, input_masks, args, kwargs)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
    outputs = call_fn(inputs, *args, **kwargs)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:

    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call  *
        outputs = self.graph.train(inputs, pre_proc_targets)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train  *
        return self(wrap((inputs, targets)))
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call  *
        return self._graph(inputs_targets)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call  *
        outputs = self.build_train_graph(inputs, targets, training=training)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph  *
        return self.build_graph(inputs, training)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph  *
        last_layer_output = layer(last_layer_output)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call  *
        return self._call(inputs, **kwargs)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call  *
        return self.lstm(inputs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__  **
        return super(Bidirectional, self).__call__(inputs, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call
        initial_state=forward_state, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__
        return super(RNN, self).__call__(inputs, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call
        inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs
        initial_state = self.get_initial_state(inputs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state
        inputs=None, batch_size=batch_size, dtype=dtype)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state
        self, inputs, batch_size, dtype))
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell
        return _generate_zero_filled_state(batch_size, cell.state_size, dtype)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state
        return nest.map_structure(create_zeros, state_size)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure
        structure[0], [func(*x) for x in entries],
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 <listcomp>
        structure[0], [func(*x) for x in entries],
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3000 create_zeros
        return array_ops.zeros(init_state_size, dtype=dtype)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2819 wrapped
        tensor = fun(*args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2868 zeros
        output = _constant_if_small(zero, shape, dtype, name)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:6 prod
        
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3052 prod
        keepdims=keepdims, initial=initial, where=where)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:86 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:855 __array__
        " a NumPy call, which is not supported".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

CRITICAL 2022-04-26 18:48:39,375             tfaip.util.logging: Uncaught exception
Traceback (most recent call last):
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/bin/calamari-train", line 8, in <module>
    sys.exit(run())
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 17, in run
    main(parse_args())
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main
    return trainer.train()
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train
    last_logs = super().train(callbacks=callbacks)
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper
    return func(*args, **kwargs)
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train
    training_graph_only=self._training_graph_only,
  File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training
    self._keras_train_model((real_inputs, wrapped_targets, real_meta))
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__
    input_list)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call
    inputs, input_masks, args, kwargs)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
    outputs = call_fn(inputs, *args, **kwargs)
  File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:

    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call  *
        outputs = self.graph.train(inputs, pre_proc_targets)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train  *
        return self(wrap((inputs, targets)))
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call  *
        return self._graph(inputs_targets)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call  *
        outputs = self.build_train_graph(inputs, targets, training=training)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph  *
        return self.build_graph(inputs, training)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph  *
        last_layer_output = layer(last_layer_output)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call  *
        return self._call(inputs, **kwargs)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call  *
        return self.lstm(inputs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__  **
        return super(Bidirectional, self).__call__(inputs, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call
        initial_state=forward_state, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__
        return super(RNN, self).__call__(inputs, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call
        inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs
        initial_state = self.get_initial_state(inputs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state
        inputs=None, batch_size=batch_size, dtype=dtype)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state
        self, inputs, batch_size, dtype))
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell
        return _generate_zero_filled_state(batch_size, cell.state_size, dtype)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state
        return nest.map_structure(create_zeros, state_size)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure
        structure[0], [func(*x) for x in entries],
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 <listcomp>
        structure[0], [func(*x) for x in entries],
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3000 create_zeros
        return array_ops.zeros(init_state_size, dtype=dtype)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2819 wrapped
        tensor = fun(*args, **kwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2868 zeros
        output = _constant_if_small(zero, shape, dtype, name)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:6 prod
        
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3052 prod
        keepdims=keepdims, initial=initial, where=where)
    /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:86 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:855 __array__
        " a NumPy call, which is not supported".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

Probably the same as #314.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
bertskycommented, Apr 27, 2022

For the sake of completeness:

Here is the SO issue that brought me on the right track.

And this is what I had to do ultimately:

pip install --no-use-pep517 numpy==1.19.5
1reaction
bertskycommented, Apr 27, 2022

Ah, found the culprit: It seems that TF has a soft dependency on certain Numpy versions. In my case, downgrading to numpy==1.19.5 for tensorflow==2.4.1 helped avoid the above error.

(I have now worked myself up to a problem with warmstarting… coming up.)

Thanks for your feedback!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Cannot convert a symbolic Tensor (2nd_target:0) to a numpy ...
For me, the issue occurred when upgrading from numpy 1.19 to 1.20 and using ray 's RLlib, which uses tensorflow 2.2 internally.
Read more >
Cannot convert a symbolic Tensor (strided_slice:0) to a numpy ...
NotImplementedError : Cannot convert a symbolic Tensor (bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate ...
Read more >
Cannot convert a symbolic Tensor to a numpy array ...
I was trying to train an object detection model using the Roboflow Google Colab notebook on the link below.
Read more >
HELP: RNN Cannot convert a symbolic Tensor Error - Reddit
I am attempting to create a RNN using my own data data i collected in an excelsheet, instead of importing the data.
Read more >
NotImplementedError: Cannot convert a symbolic Tensor ...
Without any apparent changes in my code I've started getting the following error when trying to use the NengoDL simulator with my custom ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Cannot run example script on latest version of deno

Next page

training done with the version 2.1.1 not working now despite I don't change any thing