Considering adding complex DL models such as Wide & Deep Learning Model

Hi,

It is apparent that TF-Ranking is utilizing 3-layer MLP for learning to rank. So I am considering adding other complex models demonstrating abilities for recommender systems or CTR to TFR library. The Wide & Deep learning from Google 2016 uses a wide structure for memorization and a DNN for generation. I tried to add a linear combination part to _score_fn, then use both wide and deep logits to optimize for listwise-based softmax loss. I am pasting my code here. Can someone have a look and point out if I make any mistakes? Thanks!

  def _score_fn(unused_context_features, group_features, mode, unused_params,
                unused_config):
    """Defines the network to score a group of documents."""
    with tf.compat.v1.name_scope("input_layer"):
      group_input = [
          tf.compat.v1.layers.flatten(group_features[name])
          for name in sorted(example_feature_columns())
      ]
      input_layer = tf.concat(group_input, 1)
#      tf.compat.v1.summary.scalar("input_sparsity",
#                                  tf.nn.zero_fraction(input_layer))
#      tf.compat.v1.summary.scalar("input_max",
#                                  tf.reduce_max(input_tensor=input_layer))
#      tf.compat.v1.summary.scalar("input_min",
#                                  tf.reduce_min(input_tensor=input_layer))


    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
    cur_layer = input_layer
    
    cur_layer = tf.compat.v1.layers.batch_normalization(
        input_layer, training=is_training)
    
    for i, layer_width in enumerate(int(d) for d in FLAGS.hidden_layer_dims):
      cur_layer = tf.compat.v1.layers.dense(cur_layer, units=layer_width)
      cur_layer = tf.compat.v1.layers.batch_normalization(
          cur_layer, training=is_training)
      cur_layer = tf.nn.relu(cur_layer)
      
#      tf.compat.v1.summary.scalar("fully_connected_{}_sparsity".format(i),
#                                  tf.nn.zero_fraction(cur_layer))
      cur_layer = tf.compat.v1.layers.dropout(
            cur_layer, rate=FLAGS.dropout_rate, training=is_training)
      
    logits = tf.compat.v1.layers.dense(cur_layer, units=FLAGS.group_size)
    
    print("Checkpoint 1: ", logits)
    
    dnn_logits = logits
      
    ## Build linear classifier
    group_input = [
      tf.compat.v1.layers.flatten(group_features[name])
      for name in sorted(example_feature_columns())
    ]
    input_layer = tf.concat(group_input, 1)
    
    is_training = (mode == tf.estimator.ModeKeys.TRAIN)
    cur_layer = tf.compat.v1.layers.batch_normalization(
        input_layer, training=is_training)
    
    cur_layer = tf.compat.v1.layers.dense(cur_layer, units=2)
    #cur_layer = tf.compat.v1.layers.batch_normalization(
        #cur_layer, training=is_training)
    #cur_layer = tf.nn.sigmoid(cur_layer)
    
    linear_logits = tf.compat.v1.layers.dense(cur_layer, units=FLAGS.group_size)    
    

    print("Checkpoint 2: ", logits)    
    
    # Combine logits and build full model.
    if dnn_logits is not None and linear_logits is not None:
      logits = dnn_logits + linear_logits
    elif dnn_logits is not None:
      logits = dnn_logits
    else:
      logits = linear_logits
   
    if _use_multi_head():
      # Duplicate the logits for both heads.
      return {_PRIMARY_HEAD: logits, _SECONDARY_HEAD: logits}
    else:
      return logits

  return _score_fn

[LiangqunLu]

Oh I maybe mixed a little bit. My project is Learning to rank more precisely. The features in the libsvm format are item embeddings and use embeddings. But I used TF Ranking for the ranking and did not need to use matrix factorization at this moment.

[bendersky]

Closing this discussion for now. A lot of good of advice here, we may use it for future reference.