Using openvino needs to write, causes issues with read-only container

Hi, I’ve been encountering an issue when using the openvino extra arg for calling variants. When I do not use this flag, things progress as expected. When using the extra flag, both via --call_variants_extra_args "use_openvino=true" to run_deepvariant as well as the --use_openvino to call_variants, there is the following error about being unable to save the new model due to read-only system.

This is running V1.1.0 via singularity on a linux system, using the pacbio hifi model.

Instructions for updating:
Use `tf.compat.v1.graph_util.remove_training_nodes`
Traceback (most recent call last):
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 502, in <module>
    tf.compat.v1.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/absl_py/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/absl_py/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 492, in main
    use_tpu=FLAGS.use_tpu,
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 409, in call_variants
    checkpoint_path, input_fn=tf_dataset, model=model)
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/com_google_deepvariant/deepvariant/openvino_estimator.py", line 89, in __init__
    freeze_graph(model, checkpoint_path, tensor_shape)
  File "/scratch/13530063.tmpdir/Bazel.runfiles_69_p3nci/runfiles/com_google_deepvariant/deepvariant/openvino_estimator.py", line 81, in freeze_graph
    f.write(graph_def.SerializeToString())
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/lib/io/file_io.py", line 101, in write
    self._prewrite_check()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/lib/io/file_io.py", line 87, in _prewrite_check
    compat.as_bytes(self.__name), compat.as_bytes(self.__mode))
tensorflow.python.framework.errors_impl.PermissionDeniedError: model.pb; Read-only file system

I guess since the model resides in the container, it can’t be updated as singularity is read-only without sudo. Would there be some way of passing the --checkpoint "/opt/models/pacbio/model.ckpt" parameter, but allowing it to be saved elsewhere?

Thanks, Alex

[pichuan]

Hi @ASLeonard , thanks for the report. Now I think about it, I might not have tested the combination of Singularity and our use of OpenVINO together. I’ll run this through the PacBio example and confirm that I can reproduce this issue.

There are a few fixes I can think of, which you also suggested:

1. We can provide a flag that allows users to provide a pointer to the OpenVINO model file. This will require the user to run the freeze_graph code to convert the model themselves.
2. At this line: https://github.com/google/deepvariant/blob/2dbebb4d97e15d0d5fcf303a4466314b1f313208/deepvariant/openvino_estimator.py#L80 We can provide the intermediate directory to write the model.pb file in. And users can specify a directory that Singularity won’t be unhappy with.

I’ll first try to reproduce the issue, and then I’ll likely try with option 2 above. Thanks for reporting this issue. I’ll update again when I look into it more.

Also tagging @dkurt here as an FYI.

[pichuan]

Thank you @ASLeonard for sharing your current solution.

On my end I’ll continue to try to reproduce the issue. If I’m unable to, I might need to ask you to help test out my updated code after I have it. I might reach out again later on this thread. For now, I’ll close this issue. Please feel free to update with more information, or submit another issue if you have other questions.