training: Cannot convert a symbolic Tensor to a numpy array
See original GitHub issueI have Calamari 2.2.1 / Python 3.7.4 / TF 2.4.1 / Numpy 1.21.6, and attempt calamari-train in a (Power9-based) HPC environment.
Unfortunately, I get the following exception:
stacktrace
INFO 2022-04-26 18:48:37,860 tfaip.trainer.trainer: Setting samples per epoch relative to dataset size with a factor of 1.
INFO 2022-04-26 18:48:37,860 tfaip.trainer.trainer: Set steps per epoch to 18
INFO 2022-04-26 18:48:37,874 tfaip.scenario.scenariobase: Building training keras model
INFO 2022-04-26 18:48:37,881 tfaip.scenario.scenariobase: Attempting to set no train scope
INFO 2022-04-26 18:48:37,881 tfaip.scenario.scenariobase: Compiling training model including optimization
INFO 2022-04-26 18:48:37,892 tfaip.scenario.scenariobase: Interference on the input shapes to build the graph.
CRITICAL 2022-04-26 18:48:39,339 tfaip.util.logging: Uncaught exception
Traceback (most recent call last):
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main
return trainer.train()
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train
last_logs = super().train(callbacks=callbacks)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper
return func(*args, **kwargs)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train
training_graph_only=self._training_graph_only,
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training
self._keras_train_model((real_inputs, wrapped_targets, real_meta))
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__
input_list)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call
inputs, input_masks, args, kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call *
outputs = self.graph.train(inputs, pre_proc_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train *
return self(wrap((inputs, targets)))
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call *
return self._graph(inputs_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call *
outputs = self.build_train_graph(inputs, targets, training=training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph *
return self.build_graph(inputs, training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph *
last_layer_output = layer(last_layer_output)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call *
return self._call(inputs, **kwargs)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call *
return self.lstm(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__ **
return super(Bidirectional, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call
initial_state=forward_state, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__
return super(RNN, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call
inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs
initial_state = self.get_initial_state(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state
inputs=None, batch_size=batch_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state
self, inputs, batch_size, dtype))
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell
return _generate_zero_filled_state(batch_size, cell.state_size, dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state
return nest.map_structure(create_zeros, state_size)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 <listcomp>
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3000 create_zeros
return array_ops.zeros(init_state_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
return target(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2819 wrapped
tensor = fun(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2868 zeros
output = _constant_if_small(zero, shape, dtype, name)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
if np.prod(shape) < 1000:
<__array_function__ internals>:6 prod
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3052 prod
keepdims=keepdims, initial=initial, where=where)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:86 _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:855 __array__
" a NumPy call, which is not supported".format(self.name))
NotImplementedError: Cannot convert a symbolic Tensor (root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
CRITICAL 2022-04-26 18:48:39,375 tfaip.util.logging: Uncaught exception
Traceback (most recent call last):
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/bin/calamari-train", line 8, in <module>
sys.exit(run())
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 17, in run
main(parse_args())
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main
return trainer.train()
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train
last_logs = super().train(callbacks=callbacks)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper
return func(*args, **kwargs)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train
training_graph_only=self._training_graph_only,
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training
self._keras_train_model((real_inputs, wrapped_targets, real_meta))
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__
input_list)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call
inputs, input_masks, args, kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call *
outputs = self.graph.train(inputs, pre_proc_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train *
return self(wrap((inputs, targets)))
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call *
return self._graph(inputs_targets)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call *
outputs = self.build_train_graph(inputs, targets, training=training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph *
return self.build_graph(inputs, training)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph *
last_layer_output = layer(last_layer_output)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call *
return self._call(inputs, **kwargs)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call *
return self.lstm(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__ **
return super(Bidirectional, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call
initial_state=forward_state, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__
return super(RNN, self).__call__(inputs, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__
outputs = call_fn(inputs, *args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call
inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs
initial_state = self.get_initial_state(inputs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state
inputs=None, batch_size=batch_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state
self, inputs, batch_size, dtype))
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell
return _generate_zero_filled_state(batch_size, cell.state_size, dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state
return nest.map_structure(create_zeros, state_size)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 <listcomp>
structure[0], [func(*x) for x in entries],
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3000 create_zeros
return array_ops.zeros(init_state_size, dtype=dtype)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
return target(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2819 wrapped
tensor = fun(*args, **kwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2868 zeros
output = _constant_if_small(zero, shape, dtype, name)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:2804 _constant_if_small
if np.prod(shape) < 1000:
<__array_function__ internals>:6 prod
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3052 prod
keepdims=keepdims, initial=initial, where=where)
/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/numpy/core/fromnumeric.py:86 _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:855 __array__
" a NumPy call, which is not supported".format(self.name))
NotImplementedError: Cannot convert a symbolic Tensor (root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
Probably the same as #314.
Issue Analytics
- State:
- Created a year ago
- Comments:7 (7 by maintainers)
Top Results From Across the Web
Cannot convert a symbolic Tensor (2nd_target:0) to a numpy ...
For me, the issue occurred when upgrading from numpy 1.19 to 1.20 and using ray 's RLlib, which uses tensorflow 2.2 internally.
Read more >Cannot convert a symbolic Tensor (strided_slice:0) to a numpy ...
NotImplementedError : Cannot convert a symbolic Tensor (bidirectional/forward_lstm/strided_slice:0) to a numpy array. This error may indicate ...
Read more >Cannot convert a symbolic Tensor to a numpy array ...
I was trying to train an object detection model using the Roboflow Google Colab notebook on the link below.
Read more >HELP: RNN Cannot convert a symbolic Tensor Error - Reddit
I am attempting to create a RNN using my own data data i collected in an excelsheet, instead of importing the data.
Read more >NotImplementedError: Cannot convert a symbolic Tensor ...
Without any apparent changes in my code I've started getting the following error when trying to use the NengoDL simulator with my custom ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
For the sake of completeness:
Here is the SO issue that brought me on the right track.
And this is what I had to do ultimately:
Ah, found the culprit: It seems that TF has a soft dependency on certain Numpy versions. In my case, downgrading to numpy==1.19.5 for tensorflow==2.4.1 helped avoid the above error.
(I have now worked myself up to a problem with warmstarting… coming up.)
Thanks for your feedback!