Problems with CohenKappa metrics

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• TensorFlow version and how it was installed (source or binary): 2.1.0
• TensorFlow-Addons version and how it was installed (source or binary): 0.7.0
• Python version: 3.6.9
• Is GPU used? (yes/no): yes

Describe the bug

Hello, I would to use tfa.metrics.CohenKappa from tensorflow_addons. I have a problem when I wanted to use it. I have a function where I create a basic convolution network, and I would like to use this metrics.

However, when I do that, it raised an exception

ValueError: Number of samples in y_true and y_pred are different

So I checked in the code, and it’s seam that the shape of the two Tensor are the not the same :

Tensor("Cast:0", shape=(None, None), dtype=int64) Tensor("Cast_1:0", shape=(None, 5), dtype=int64)

I wanted to know, how I can I precise the shape of the y_pred in order to have the same shape as the y_true.

Code to reproduce the issue

def convolution(categories=5, shape_x=224, shape_y=224, channels=3):
    model = tf.keras.models.Sequential([
      tf.keras.layers.Conv2D(10, kernel_size=(5, 5), strides=(1, 1), activation=tf.nn.relu, use_bias=True, input_shape=(shape_x, shape_y, channels)),
      tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(1, 1), padding='valid'),
      tf.keras.layers.Conv2D(10, kernel_size=(5, 5), strides=(1, 1), activation=tf.nn.relu, use_bias=True),
      tf.keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(1, 1), padding='valid'),
      tf.keras.layers.Flatten(),
      tf.keras.layers.Dense(128, activation=tf.nn.relu),
      tf.keras.layers.Dense(categories, activation=tf.nn.softmax)
    ])
    model.compile(optimizer='adam', loss='mse', metrics=[tfa.metrics.CohenKappa(num_classes=5)])
    return model

Other info / logs

File “/home/rere/Project/Aptos/aptos2019-blindness-detection/model.py”, line 38, in convolution model.compile(optimizer=‘adam’, loss=‘mse’, metrics=[tfa.metrics.CohenKappa(num_classes=5)]) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/base.py”, line 457, in _method_wrapper result = method(self, *args, **kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py”, line 439, in compile masks=self._prepare_output_masks()) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py”, line 2004, in _handle_metrics target, output, output_mask)) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py”, line 1955, in _handle_per_output_metrics metric_fn, y_true, y_pred, weights=weights, mask=mask) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_utils.py”, line 1155, in call_metric_function return metric_fn(y_true, y_pred, sample_weight=weights) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py”, line 196, in call replica_local_fn, *args, **kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py”, line 1135, in call_replica_local_fn return fn(*args, **kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py”, line 179, in replica_local_fn update_op = self.update_state(*args, **kwargs) # pylint: disable=not-callable File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/metrics_utils.py”, line 76, in decorated update_op = update_state_fn(*args, **kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py”, line 568, in call result = self._call(*args, **kwds) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py”, line 615, in _call self._initialize(args, kwds, add_initializers_to=initializers) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py”, line 497, in _initialize *args, **kwds)) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 2389, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 2703, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 2593, in _create_graph_function capture_by_value=self._capture_by_value), File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py”, line 978, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py”, line 439, in wrapped_fn return weak_wrapped_fn().wrapped(*args, **kwds) File “/home/rere/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py”, line 968, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code:

/home/rere/.local/lib/python3.6/site-packages/tensorflow_addons/metrics/cohens_kappa.py:122 update_state  *
    raise ValueError(

ValueError: Number of samples in `y_true` and `y_pred` are different

Thank you in advance for any help 😃

[AakashKumarNain]

Thanks @facaiy for the review. I discussed on this with Francois as well. I will try to make it more simple in the coming weeks.

[facaiy]

@AakashKumarNain Sorry for the delay, Aakash. Thanks for your detailed RFC, which looks really great. As said before, is it possible to create a metric for every case mentioned by you, for example: CohenKappa, BinaryCohenKappa, CategoricalCohenKappa etc (refer to Accuracy, BinaryAccuracy, CategoricalAccuracy, … ). What do you think? cc @WindQAQ @seanpmorgan