Load best exporter using Keras implementation

Hello,

I’m trying to load the best model trained using checkpoints. Here is my code to save checkpoints:

mcp_save = tf.keras.callbacks.ModelCheckpoint(
    "./checkpoint",
    monitor="metric/proba_diff",
    verbose=1,
    save_best_only=True,
    save_weights_only=False,
    mode="max",
    save_freq="epoch",
    options=None
)


ranker.fit(
    x=train_dataset,
    y=None,
    epochs=epochs,
    verbose=2,
    callbacks=[mcp_save, tensorboard_callback],
    validation_data=eval_dataset,
    shuffle=True,
    steps_per_epoch=1000,
    validation_steps=None,
    validation_freq=1,
)

After the training, when I want to evaluate the model saved I load it like that:

model = tf.saved_model.load(model_export_path)
predict_fn = model.signatures[signature]
predictions = batched_ds.map(lambda x: predict_fn(**{input_arg: x}), num_parallel_calls=nb_threads)

but it doesn’t actually load the best exporter, or update the weights from the folder checkpoint.

I tried to copy the data from checkpoint to the model folder but then I got this error:

raceback (most recent call last):
  File "/home/till/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/till/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/till/tfranking_recommender/src/tfranking_recommender/__main__.py", line 4, in <module>
    cli()
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/till/tfranking_recommender/src/tfranking_recommender/cli/evaluate.py", line 77, in evaluate
    tf.compat.v1.app.run(main=main)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/till/tfranking_recommender/src/tfranking_recommender/cli/evaluate.py", line 53, in main
    nb_threads=click_config.nb_threads)
  File "/home/till/tfranking_recommender/src/tfranking_recommender/eval/evaluate.py", line 24, in evaluate_raw_elwc
    predictions = batched_ds.map(lambda x: predict_fn(**{input_arg: x}), num_parallel_calls=nb_threads)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1805, in map
    return MapDataset(self, map_func, preserve_cardinality=True)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4207, in __init__
    use_legacy_function=use_legacy_function)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3525, in __init__
    self._function = wrapper_fn.get_concrete_function()
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3052, in get_concrete_function
    *args, **kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3019, in _get_concrete_function_garbage_collected
    graph_function, _ = self._maybe_define_function(args, kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3361, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3206, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 990, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3518, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3453, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "/home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    /home/till/tfranking_recommender/src/tfranking_recommender/eval/evaluate.py:24 None  *
        lambda x: predict_fn(**{input_arg: x}), num_parallel_calls=nb_threads)
    /home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py:1669 __call__  **
        return self._call_impl(args, kwargs)
    /home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py:1685 _call_impl
        raise structured_err
    /home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py:1679 _call_impl
        cancellation_manager)
    /home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py:1756 _call_with_structured_signature
        self._structured_signature_check_missing_args(args, kwargs)
    /home/till/tfranking_recommender/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py:1777 _structured_signature_check_missing_args
        ", ".join(sorted(missing_arguments))))

    TypeError: signature_wrapper(*, args_0_180, args_0_109, args_0_95, args_0_224, args_0_96, args_0_139, args_0_20, args_0_242, args_0_74, args_0_66, args_0_98, args_0_61, args_0_39, args_0_229, args_0_76, args_0_151, args_0_166, args_0_49, args_0_170, args_0_46, args_0_149, args_0_10, args_0_111, args_0_144, args_0_168, args_0_11, args_0_97, args_0_159, args_0_68, args_0_55, a
rgs_0_36, args_0_196, args_0_58, args_0_62, args_0_71, args_0_54, args_0_70, args_0_8, args_0_103, args_0_167, args_0_153, args_0_215, args_0_177, args_0_38, args_0_183, args_0_192, args_0_187, args_0_251, args_0_213, args_0_239, args_0_223, args_0_100, args_0_174, args_0_137, args_0_65, args_0_13, args_0_135, args_0_52, args_0_245, args_0_93, args_0_60, args_0_148, args_0_89, a
rgs_0_236, args_0_43, args_0_146, args_0_133, args_0_136, args_0_5, args_0_150, args_0_182, args_0_106, args_0_231, args_0_47, args_0_16, args_0_246, args_0_179, args_0_29, args_0_118, args_0_3, args_0_56, args_0_156, args_0_145, args_0_26, args_0_178, args_0_127, args_0_41, args_0_234, args_0_22, args_0_209, args_0_28, args_0_63, args_0_53, args_0_59, args_0_81, args_0_128, arg
s_0_175, args_0_6, args_0_181, args_0_216, args_0_79, args_0_37, args_0_108, args_0_201, args_0_155, args_0_7, args_0_4, args_0_238, args_0_165, args_0_113, args_0_176, args_0_32, args_0_194, args_0_249, args_0_83, args_0_228, args_0_141, args_0_126, args_0_253, args_0_101, args_0_257, args_0_84, args_0_94, args_0_57, args_0_33, args_0_77, args_0_217, args_0_42, args_0_15, args_
0_35, args_0_237, args_0_125, args_0_102, args_0_132, args_0_186, args_0_211, args_0_154, args_0_202, args_0_40, args_0_17, args_0_121, args_0_163, args_0_222, args_0_23, args_0_24, args_0_92, args_0_85, args_0_199, args_0_88, args_0_104, args_0_25, args_0_117, args_0, args_0_162, args_0_252, args_0_193, args_0_200, args_0_69, args_0_158, args_0_131, args_0_27, args_0_250, args_
0_116, args_0_190, args_0_247, args_0_110, args_0_122, args_0_221, args_0_78, args_0_31, args_0_21, args_0_207, args_0_64, args_0_161, args_0_191, args_0_254, args_0_114, args_0_240, args_0_51, args_0_75, args_0_105, args_0_86, args_0_140, args_0_112, args_0_50, args_0_218, args_0_172, args_0_225, args_0_243, args_0_72, args_0_123, args_0_157, args_0_204, args_0_208, args_0_152,
 args_0_160, args_0_2, args_0_195, args_0_91, args_0_205, args_0_9, args_0_14, args_0_248, args_0_232, args_0_1, args_0_219, args_0_226, args_0_134, args_0_18, args_0_184, args_0_87, args_0_45, args_0_220, args_0_90, args_0_214, args_0_206, args_0_210, args_0_173, args_0_198, args_0_233, args_0_147, args_0_256, args_0_203, args_0_169, args_0_82, args_0_227, args_0_30, args_0_244
, args_0_197, args_0_188, args_0_129, args_0_189, args_0_142, args_0_124, args_0_107, args_0_164, args_0_44, args_0_115, args_0_67, args_0_120, args_0_235, args_0_80, args_0_99, args_0_143, args_0_48, args_0_119, args_0_138, args_0_241, args_0_171, args_0_130, args_0_12, args_0_34, args_0_185, args_0_212, args_0_73, args_0_255, args_0_19, args_0_230) missing required arguments:
args_0, args_0_1, args_0_10, args_0_100, args_0_101, args_0_102, args_0_103, args_0_104, args_0_105, args_0_106, args_0_107, args_0_108, args_0_109, args_0_11, args_0_110, args_0_111, args_0_112, args_0_113, args_0_114, args_0_115, args_0_116, args_0_117, args_0_118, args_0_119, args_0_12, args_0_120, args_0_121, args_0_122, args_0_123, args_0_124, args_0_125, args_0_126, args_0
_127, args_0_128, args_0_129, args_0_13, args_0_130, args_0_131, args_0_132, args_0_133, args_0_134, args_0_135, args_0_136, args_0_137, args_0_138, args_0_139, args_0_14, args_0_140, args_0_141, args_0_142, args_0_143, args_0_144, args_0_145, args_0_146, args_0_147, args_0_148, args_0_149, args_0_15, args_0_150, args_0_151, args_0_152, args_0_153, args_0_154, args_0_155, args_0
_156, args_0_157, args_0_158, args_0_159, args_0_16, args_0_160, args_0_161, args_0_162, args_0_163, args_0_164, args_0_165, args_0_166, args_0_167, args_0_168, args_0_169, args_0_17, args_0_170, args_0_171, args_0_172, args_0_173, args_0_174, args_0_175, args_0_176, args_0_177, args_0_178, args_0_179, args_0_18, args_0_180, args_0_181, args_0_182, args_0_183, args_0_184, args_0
_185, args_0_186, args_0_187, args_0_188, args_0_189, args_0_19, args_0_190, args_0_191, args_0_192, args_0_193, args_0_194, args_0_195, args_0_196, args_0_197, args_0_198, args_0_199, args_0_2, args_0_20, args_0_200, args_0_201, args_0_202, args_0_203, args_0_204, args_0_205, args_0_206, args_0_207, args_0_208, args_0_209, args_0_21, args_0_210, args_0_211, args_0_212, args_0_2
13, args_0_214, args_0_215, args_0_216, args_0_217, args_0_218, args_0_219, args_0_22, args_0_220, args_0_221, args_0_222, args_0_223, args_0_224, args_0_225, args_0_226, args_0_227, args_0_228, args_0_229, args_0_23, args_0_230, args_0_231, args_0_232, args_0_233, args_0_234, args_0_235, args_0_236, args_0_237, args_0_238, args_0_239, args_0_24, args_0_240, args_0_241, args_0_2
42, args_0_243, args_0_244, args_0_245, args_0_246, args_0_247, args_0_248, args_0_249, args_0_25, args_0_250, args_0_251, args_0_252, args_0_253, args_0_254, args_0_255, args_0_256, args_0_257, args_0_26, args_0_27, args_0_28, args_0_29, args_0_3, args_0_30, args_0_31, args_0_32, args_0_33, args_0_34, args_0_35, args_0_36, args_0_37, args_0_38, args_0_39, args_0_4, args_0_40, a
rgs_0_41, args_0_42, args_0_43, args_0_44, args_0_45, args_0_46, args_0_47, args_0_48, args_0_49, args_0_5, args_0_50, args_0_51, args_0_52, args_0_53, args_0_54, args_0_55, args_0_56, args_0_57, args_0_58, args_0_59, args_0_6, args_0_60, args_0_61, args_0_62, args_0_63, args_0_64, args_0_65, args_0_66, args_0_67, args_0_68, args_0_69, args_0_7, args_0_70, args_0_71, args_0_72,
args_0_73, args_0_74, args_0_75, args_0_76, args_0_77, args_0_78, args_0_79, args_0_8, args_0_80, args_0_81, args_0_82, args_0_83, args_0_84, args_0_85, args_0_86, args_0_87, args_0_88, args_0_89, args_0_9, args_0_90, args_0_91, args_0_92, args_0_93, args_0_94, args_0_95, args_0_96, args_0_97, args_0_98, args_0_99

Do you know what is the proper way to load the best exporter using Keras implementation ?

Thank you

On Ubuntu 18.04, Python 3.7.3

tensorflow==2.4.1
tensorflow-addons==0.12.1
tensorflow-datasets==4.2.0
tensorflow-estimator==2.4.0
tensorflow-hub==0.11.0
tensorflow-metadata==0.29.0
tensorflow-model-optimization==0.5.0
tensorflow-ranking==0.3.3
tensorflow-serving-api==2.4.1
tensorflow-transform==0.29.0

[lyyanlely]

It appears to me that the error is due to that model.signatures cannot be loaded from checkpoint. You need to export the model with tf.saved_model.save to define signatures or with model.save as in our built-in pipeline.

Note that checkpoint has model graph and weights but not input/output definition for signatures.

[tillwf]

Thank you very much for your help. I actually did that and it works well:

checkpoint = tf.train.latest_checkpoint("./checkpoints")
ranker.load_weights(checkpoint)
ranker.save(model_dir, save_format='tf', signatures=signatures)