Model.fit call generates error ValueError: Shape must be rank 2 but is rank 3

I get the below in my TFRS prototype where I build a model with user ID’s and item ID’s, no other features.

Does this error indicate a mismatch of batch sizes? but of what and what? In the ‘retrieval’ sample, I don’t see a match of batch size e.g. between the train dataset and any other datasets. Any clues, would appreciate it.

WARNING:tensorflow:Model was constructed with shape (None, 1000) for input KerasTensor(type_spec=TensorSpec(shape=(None, 1000), dtype=tf.string, name='string_lookup_1_input'), name='string_lookup_1_input', description="created by layer 'string_lookup_1_input'"), but it was called on an input with incompatible shape (None,).
WARNING:tensorflow:From /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version.
Instructions for updating:
The `validate_indices` argument has no effect. Indices are always validated on CPU and never validated on GPU.
Traceback (most recent call last):
  File "/mnt/tmp/spark-23c1419e-4a5c-4ec7-a86f-1f6f23be73d3/recsys_tfrs_songs.py", line 90, in <module>
    main(sys.argv)
  File "/mnt/tmp/spark-23c1419e-4a5c-4ec7-a86f-1f6f23be73d3/recsys_tfrs_songs.py", line 61, in main
    model_maker.train_and_evaluate(model, NUM_TRAIN_EPOCHS)
  File "/mnt/tmp/spark-23c1419e-4a5c-4ec7-a86f-1f6f23be73d3/recsys-deps.zip/recommender_system/recsys_tf/recsys_tfrs_model.py", line 151, in train_and_evaluate
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1152, in fit
    tmp_logs = self.train_function(iterator)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 867, in __call__
    result = self._call(*args, **kwds)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 911, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 749, in _initialize
    *args, **kwds))
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3045, in _get_concrete_function_internal_garbage_collected
    graph_function, _ = self._maybe_define_function(args, kwargs)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3439, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3284, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 998, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 657, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 985, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:847 train_function  *
        return step_function(self, iterator)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow_recommenders/tasks/retrieval.py:157 call  *
        update_op = self._factorized_metrics.update_state(query_embeddings,
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow_recommenders/metrics/factorized_top_k.py:83 update_state  *
        top_k_predictions, _ = self._candidates(query_embeddings, k=self._k)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow_recommenders/layers/factorized_top_k.py:224 top_k  *
        joined_scores = tf.concat([state_scores, x_scores], axis=1)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:206 wrapper  **
        return target(*args, **kwargs)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:1768 concat
        return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:1208 concat_v2
        "ConcatV2", values=values, axis=axis, name=name)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:750 _apply_op_helper
        attrs=attr_protos, op_def=op_def)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py:600 _create_op_internal
        compute_device)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:3554 _create_op_internal
        op_def=op_def)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:2031 __init__
        control_input_ops, op_def)
    /home/hadoop/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1872 _create_c_op
        raise ValueError(str(e))

    ValueError: Shape must be rank 2 but is rank 3 for '{{node concat}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32](args_0, args_2, concat/axis)' with input shapes: [?,0], [?,?,?], [].

[dgoldenberg-audiomack]

This can be closed although I’d venture to say TF should fix this super-confusing message.

[dgoldenberg-audiomack]

@erikmajlath It’s something along these lines for me too. I can tell that when I run the retrieval sample:

the movies ds is shaped like this:

movies = movies.map(lambda x: x["movie_title"])
<MapDataset shapes: (), types: tf.string>

the candidates ds is shaped like this:

cands = movies.batch(128).map(movie_model)
metrics = tfrs.metrics.FactorizedTopK(candidates=cands)

cands is:
<MapDataset shapes: (None, 32), types: tf.float32>

However, when I run my code, my items dataset is

cands = items_ds.batch(128).map(item_model)
metrics = tfrs.metrics.FactorizedTopK(candidates=cands)

candidates:
<MapDataset shapes: (None, 1000, 32), types: tf.float32>

I’m getting that extra 1000 when I invoke

items_ds = tf.data.experimental.make_csv_dataset(
    local_file_list, column_names=["item_id"], batch_size=1000, num_parallel_reads=50, sloppy=True,
)

I’m thinking to try and invoke it without setting the batch size so this 1K doesn’t get wired in (hmm, batch size is required there). If not maybe run_eagerly=True as you were saying…