error happened at training

Thanks for sharing the code. Error happened when training is as follows, `Epoch 1/15 702/703 [============================>.] - ETA: 0s - loss: 62.9782 - f1: 84.33 703/703 [==============================] - 78s - loss: 62.8931
Epoch 2/15 702/703 [============================>.] - ETA: 0s - loss: 60.5871 - f1: 87.10 703/703 [==============================] - 80s - loss: 60.5013
Epoch 3/15 702/703 [============================>.] - ETA: 0s - loss: 59.9166 - f1: 89.38 703/703 [==============================] - 78s - loss: 59.8314
Epoch 4/15 702/703 [============================>.] - ETA: 0s - loss: 59.1642 - f1: 89.96 703/703 [==============================] - 77s - loss: 59.0806
Epoch 5/15 702/703 [============================>.] - ETA: 0s - loss: 59.6137 - f1: 90.65 703/703 [==============================] - 78s - loss: 59.5299
Epoch 6/15 702/703 [============================>.] - ETA: 0s - loss: 59.4186 - f1: 90.61 703/703 [==============================] - 78s - loss: 59.3342
Epoch 7/15 702/703 [============================>.] - ETA: 0s - loss: 59.5612 - f1: 91.40 703/703 [==============================] - 78s - loss: 59.4771
Epoch 8/15 702/703 [============================>.] - ETA: 0s - loss: 59.3047 - f1: 91.13 703/703 [==============================] - 78s - loss: 59.2204
Epoch 9/15 702/703 [============================>.] - ETA: 0s - loss: 59.5008

UnimplementedError Traceback (most recent call last) <ipython-input-6-461a48430fd1> in <module>() 1 trainer = anago.Trainer(model_config, training_config, checkpoint_path=LOG_ROOT, save_path=SAVE_ROOT, 2 preprocessor=p, embeddings=embeddings) ----> 3 trainer.train(x_train, y_train, x_valid, y_valid)

/home/hongzhi/wp/anago/anago/trainer.pyc in train(self, x_train, y_train, x_valid, y_valid) 52 steps_per_epoch=train_steps, 53 epochs=self.training_config.max_epoch, —> 54 callbacks=callbacks) 55 56 # Save the model

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs) 85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ’ + signature, stacklevel=2) —> 87 return func(*args, **kwargs) 88 wrapper._original_function = func 89 return wrapper

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch) 2040 outs = self.train_on_batch(x, y, 2041 sample_weight=sample_weight, -> 2042 class_weight=class_weight) 2043 2044 if not isinstance(outs, list):

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in train_on_batch(self, x, y, sample_weight, class_weight) 1760 ins = x + y + sample_weights 1761 self._make_train_function() -> 1762 outputs = self.train_function(ins) 1763 if len(outputs) == 1: 1764 return outputs[0]

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc in call(self, inputs) 2271 updated = session.run(self.outputs + [self.updates_op], 2272 feed_dict=feed_dict, -> 2273 **self.session_kwargs) 2274 return updated[:len(self.outputs)] 2275

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata) 787 try: 788 result = self._run(None, fetches, feed_dict, options_ptr, –> 789 run_metadata_ptr) 790 if run_metadata: 791 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata) 995 if final_fetches or final_targets: 996 results = self._do_run(handle, final_targets, final_fetches, –> 997 feed_dict_string, options, run_metadata) 998 else: 999 results = []

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata) 1130 if handle is None: 1131 return self._do_call(_run_fn, self._session, feed_dict, fetch_list, -> 1132 target_list, options, run_metadata) 1133 else: 1134 return self._do_call(_prun_fn, self._session, handle, feed_dict,

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args) 1150 except KeyError: 1151 pass -> 1152 raise type(e)(node_def, op, message) 1153 1154 def _extend_graph(self):

UnimplementedError: TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays. [[Node: chain_crf_2/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=[“loc:@chain_crf_2/TensorArray”], dtype=DT_FLOAT, element_shape=[?,10], _device=“/job:localhost/replica:0/task:0/gpu:0”](chain_crf_2/TensorArray, chain_crf_2/TensorArrayStack/range, chain_crf_2/while/Exit_1)]] [[Node: training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch/_887 = _Recvclient_terminated=false, recv_device=“/job:localhost/replica:0/task:0/cpu:0”, send_device=“/job:localhost/replica:0/task:0/gpu:0”, send_device_incarnation=1, tensor_name=“edge_5628_training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch”, tensor_type=DT_FLOAT, _device=“/job:localhost/replica:0/task:0/cpu:0”]]

Caused by op u’chain_crf_2/TensorArrayStack/TensorArrayGatherV3’, defined at: File “/home/hongzhi/anaconda2/lib/python2.7/runpy.py”, line 174, in _run_module_as_main “main”, fname, loader, pkg_name) File “/home/hongzhi/anaconda2/lib/python2.7/runpy.py”, line 72, in _run_code exec code in run_globals File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/main.py”, line 3, in <module> app.launch_new_instance() File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py”, line 653, in launch_instance app.start() File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py”, line 474, in start ioloop.IOLoop.instance().start() File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py”, line 162, in start super(ZMQIOLoop, self).start() File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tornado/ioloop.py”, line 887, in start handler_func(fd_obj, events) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py”, line 275, in null_wrapper return fn(*args, **kwargs) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py”, line 440, in _handle_events self._handle_recv() File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py”, line 472, in _handle_recv self._run_callback(callback, msg) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py”, line 414, in _run_callback callback(*args, **kwargs) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py”, line 275, in null_wrapper return fn(*args, **kwargs) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py”, line 276, in dispatcher return self.dispatch_shell(stream, msg) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py”, line 228, in dispatch_shell handler(stream, idents, msg) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py”, line 390, in execute_request user_expressions, allow_stdin) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/ipkernel.py”, line 196, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/zmqshell.py”, line 501, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2717, in run_cell interactivity=interactivity, compiler=compiler, result=result) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2827, in run_ast_nodes if self.run_code(code, result): File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py”, line 2881, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File “<ipython-input-6-461a48430fd1>”, line 3, in <module> trainer.train(x_train, y_train, x_valid, y_valid) File “anago/trainer.py”, line 39, in train model = SeqLabeling(self.model_config, self.embeddings, len(self.preprocessor.vocab_tag)) File “anago/models.py”, line 83, in init pred = self.crf(x) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py”, line 602, in call output = self.call(inputs, **kwargs) File “anago/layers.py”, line 314, in call y_pred = viterbi_decode(x, self.U, self.b_start, self.b_end, mask) File “anago/layers.py”, line 106, in viterbi_decode mask) File “anago/layers.py”, line 147, in _forward last, values, _ = K.rnn(_forward_step, inputs, initial_states) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py”, line 2551, in rnn outputs = output_ta.stack() File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/tensor_array_ops.py”, line 334, in stack return self.gather(math_ops.range(0, self.size()), name=name) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/tensor_array_ops.py”, line 360, in gather element_shape=element_shape) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py”, line 1814, in _tensor_array_gather_v3 element_shape=element_shape, name=name) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py”, line 767, in apply_op op_def=op_def) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py”, line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File “/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py”, line 1269, in init self._traceback = _extract_stack()

UnimplementedError (see above for traceback): TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays. [[Node: chain_crf_2/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=[“loc:@chain_crf_2/TensorArray”], dtype=DT_FLOAT, element_shape=[?,10], _device=“/job:localhost/replica:0/task:0/gpu:0”](chain_crf_2/TensorArray, chain_crf_2/TensorArrayStack/range, chain_crf_2/while/Exit_1)]] [[Node: training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch/_887 = _Recvclient_terminated=false, recv_device=“/job:localhost/replica:0/task:0/cpu:0”, send_device=“/job:localhost/replica:0/task:0/gpu:0”, send_device_incarnation=1, tensor_name=“edge_5628_training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch”, tensor_type=DT_FLOAT, _device=“/job:localhost/replica:0/task:0/cpu:0”]] My environmentsanago (0.0.1) Keras (2.0.8) lazy-object-proxy (1.2.1) tensorflow-gpu (1.2.0) tensorflow-tensorboard (0.1.6) ` Any idea to fix it is appreciated, thanks.

[michaelwangtd]

error blow： @Hironsan thanks! tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays. [[Node: chain_crf_1/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=[“loc:@chain_crf_1/TensorArray”], dtype=DT_FLOAT, element_shape=[?,10], _device=“/job:localhost/replica:0/task:0/cpu:0”](chain_crf_1/TensorArray, chain_crf_1/TensorArrayStack/range, chain_crf_1/while/Exit_1)]]

2017-10-24 12:29:20.151720: W C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\35\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: Tried to read from index -1 but array size is: 0

windows10 tensorflow：1.3.0 tensorboard：0.1.8 keras：2.0.5

[Hironsan]

I updated anaGo, try it again.

$ pip install -U anago