Caching in tfx and kfp

I’m confused about caching when running a TFX pipeline on KFP.

• I have KFP 1.0.4 deployed via GCP AI platform and I’ve been using both TFX 0.25 and 0.26 when trying this.
• I created my pipeline with enable_cache=False.
• I ran it once and each component runs as expected and produces the expected artifacts (only checked the artifact buckets and not the sql database).
• I ran it again with no change in inputs or parameters and it’s now using cached results even though I disabled the cache. The following images show the logs for the second run where I would have expected them to run again.
• I repeated this for an entirely new deployment of KFP for both cases.

With TFX 0.25

With TFX 0.26

Questions:

• Why is the enable_cache=False not respected?
• KFP documentation mentions that its caching mechanisms should not be used for TFX pipelines. Why am I seeing a message about a cached step from KFP in the TFX 0.26 case rather than from the TFX component drivers?
• Can you enable/disable caching on a per component basis?
• Is it possible to get some logs for why a cached result was used / not used?
• Is there anyway to get TFX to also consider if the docker image has changed when determining if a cached result is invalidated?
  • I’m guessing no since it’s determined by the driver which would run inside the (updated) container?
  • Would the custom container based component handle this differently?
• Is there a better place to ask these kind of questions?

[ssoudan]

Something like this works for me:

  def dont_cache_components():
    def _dont_cache(pod_op):
      pod_op.add_pod_annotation('pipelines.kubeflow.org/max_cache_staleness', 'P0D')
      return pod_op

    return _dont_cache

  # ... 
  pipeline_op_funcs = kubeflow_dag_runner.get_default_pipeline_operator_funcs()
  pipeline_op_funcs.append(dont_cache_components())
  # ...

  runner_config = kubeflow_dag_runner.KubeflowDagRunnerConfig(
    pipeline_operator_funcs=pipeline_op_funcs,
    kubeflow_metadata_config=metadata_config,
    tfx_image=tfx_image)

[easadler]

@johnPertoft this is also a blocker for me. We will have to pivot towards vanilla kubeflow, which really isn’t that bad to use. I will miss the interactive runner, but using the kfp is pretty easy. It would be nice to have kubeflow components for each of the tfx steps. Probably will happen eventually.