Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Esoteric bug whilst deepcopy-ing

See original GitHub issue

Our training procedure involves deepcopying a model. This fails when we have a parameter with tfp.distributions.LogNormal prior that is used early in training, but is subsequently disabled. The failure is TypeError: can't pickle HashableWeakRef objects.

To reproduce

I’ve attempted to trim away all irrelevant bits from our full training pipeline. It looks like the root cause relates to sampling from tfp.distributions.LogNormal (although anecdotally, we’ve seen similar issues with some bijectors).

Minimal, reproducible example

import gpflow
import numpy as np
import tensorflow as tf
import tensorflow_probability as tfp


def boom() -> None:
    m = tf.Module()
    m.prior = tfp.distributions.LogNormal(0.0, 1.0)  # error
    # m.prior = tfp.distributions.Normal(0.0, 1.0)  # no error

    @tf.function  # no error without the decorator
    def _sample() -> tf.Tensor:
        m.prior.sample()
        # it looks like the sampling pollutes the graph - if you comment out the line above,
        # then we don't explode

    _sample()
    m.prior = None  # if you comment out this line, then we don't explode

    gpflow.utilities.deepcopy(m)


if __name__ == '__main__':
    boom()

Stack trace, or error message

  File "/home/elvijs/.PyCharmCE2019.2/config/scratches/scratch_7.py", line 39, in <module>
    boom()
  File "/home/elvijs/.PyCharmCE2019.2/config/scratches/scratch_7.py", line 35, in boom
    gpflow.utilities.deepcopy(gp_model)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/site-packages/gpflow/utilities/utilities.py", line 265, in deepcopy
    return copy.deepcopy(reset_cache_bijectors(input_module), memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 273, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 273, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 273, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 305, in _reconstruct
    key = deepcopy(key, memo)
  File "/home/elvijs/.virtualenvs/automl_res/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle HashableWeakRef objects

Expected behavior

I expected no blow-up.

System information

  • GPflow version: gpflow @ git+https://github.com/GPflow/GPflow.git@61f8d84ea170791460468ef39ce5c38d3ec20a2a (that is, I’m including the fixes in https://github.com/GPflow/GPflow/pull/1476)
  • GPflow installed from: pip install git+https://github.com/GPflow/GPflow.git@61f8d84ea170791460468ef39ce5c38d3ec20a2a#egg=gpflow,
  • TensorFlow version: 2.2.0
  • TensorFlow Probability version: 0.10.0
  • Python version: 3.7.5
  • Operating system: Ubuntu

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:14 (14 by maintainers)

github_iconTop GitHub Comments

1reaction
elvijscommented, May 21, 2020

Update: raised https://github.com/tensorflow/probability/issues/944. Closing the present issue.

1reaction
awavcommented, May 20, 2020

As a workaround, we could add a setter to gpflow’s Parameter.prior that resets the cache before overwriting it.

Please, no. Prohibiting any attempts to change the prior would be more reasonable.

@elvijs , Q: Why do you want to reset prior to None? Changing objects in-place is always a dangerous thing to do. Would be a parameters snapshot and a copy of the object a reasonable alternative for you?

Read more comments on GitHub >

github_iconTop Results From Across the Web

What is the difference between a deep copy and a shallow ...
With a deep copy, any object pointed to by the source is copied and the copy is pointed to by the destination (so...
Read more >
Issue 29897: itertools.chain behaves strangly when copied ...
When using `copy.copy` to copy an `itertools.chain` instance the results ... how pythons iterators behave when copied, deepcopied or pickled ...
Read more >
Deepcopy - Rosetta Code
Task Demonstrate how to copy data structures containing complex heterogeneous and cyclic semantics. This is often referred to as deep ...
Read more >
Deppcopying a NavigableString in a large tree can exceed the ...
I encountered a strange bug, unfortunately I was not able to build a minimum HTML code that replicates the bug. So I am...
Read more >
Chapter 8 - Common Python Gotchas
The task seems straightforward: iterate over the list's strings, and when you ... advice is to always use copy.deepcopy() : it might prevent...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

GPflow is not predicting classes

Next page

predict_y and predict_log_density support for full_cov or full_output_cov