Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pickle Error with TF2.0

See original GitHub issue

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04 (LTS)
Ray installed from (source or binary): installed with pip
Ray version: 0.7.5
Python version: 3.6
Exact command to reproduce: Execute code from https://ray.readthedocs.io/en/latest/using-ray-with-tensorflow.html

Describe the problem

I cannot use ray with Tensorflow 2.0 (stable). Even using the documentation code at https://ray.readthedocs.io/en/latest/using-ray-with-tensorflow.html throws the same pickling error as my own code:

Source code / logs

Here for quick copying is the code:

#!/usr/bin/env python
"""TODO Module Docstring."""

import numpy as np
import ray
import tensorflow as tf
from tensorflow.keras import layers


def create_keras_model():
    model = tf.keras.Sequential()
    # Adds a densely-connected layer with 64 units to the model:
    model.add(layers.Dense(64, activation="relu", input_shape=(32,)))
    # Add another:
    model.add(layers.Dense(64, activation="relu"))
    # Add a softmax layer with 10 output units:
    model.add(layers.Dense(10, activation="softmax"))

    model.compile(
        optimizer=tf.train.RMSPropOptimizer(0.01),
        loss=tf.keras.losses.categorical_crossentropy,
        metrics=[tf.keras.metrics.categorical_accuracy])
    return model


def random_one_hot_labels(shape):
    n, n_class = shape
    classes = np.random.randint(0, n_class, n)
    labels = np.zeros((n, n_class))
    labels[np.arange(n), classes] = 1
    return labels


ray.init()


@ray.remote
class Network(object):
    def __init__(self):
        self.model = create_keras_model()
        self.dataset = np.random.random((1000, 32))
        self.labels = random_one_hot_labels((1000, 10))

    def train(self):
        history = self.model.fit(self.dataset, self.labels, verbose=False)
        return history.history

    def get_weights(self):
        return self.model.get_weights()

    def set_weights(self, weights):
        # Note that for simplicity this does not handle the optimizer state.
        self.model.set_weights(weights)


NetworkActor = Network.remote()
result_object_id = NetworkActor.train.remote()
ray.get(result_object_id)

This gives a TypeError:

Traceback (most recent call last):
  File "/home/***/test.py", line 56, in <module>
    NetworkActor = Network.remote()
  File "/home/***/venv/lib/python3.6/site-packages/ray/actor.py", line 322, in remote
    return self._remote(args=args, kwargs=kwargs)
  File "/home/***/venv/lib/python3.6/site-packages/ray/actor.py", line 405, in _remote
    self._modified_class, self._actor_method_names)
  File "/home/***/venv/lib/python3.6/site-packages/ray/function_manager.py", line 578, in export_actor_class
    "class": pickle.dumps(Class),
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 1123, in dumps
    cp.dump(obj)
  File "/home/***/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 482, in dump
    return Pickler.dump(self, obj)
  File "/usr/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 875, in save_global
    self.save_dynamic_class(obj)
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 682, in save_dynamic_class
    obj=obj)
  File "/usr/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 875, in save_global
    self.save_dynamic_class(obj)
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 686, in save_dynamic_class
    save(clsdict)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 556, in save_function
    return self.save_function_tuple(obj)
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 756, in save_function_tuple
    save(state)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 556, in save_function
    return self.save_function_tuple(obj)
  File "/home/***/venv/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle.py", line 756, in save_function_tuple
    save(state)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.6/pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "/usr/lib/python3.6/pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "/usr/lib/python3.6/pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle _LazyLoader objects

Issue Analytics

State:
Created 4 years ago
Comments:11 (2 by maintainers)

Top GitHub Comments

5reactions

richardliawcommented, Oct 14, 2019

Try this?

import numpy as np
import ray
from tensorflow.keras import layers


def create_keras_model():
    import tensorflow as tf
    model = tf.keras.Sequential()
    # Adds a densely-connected layer with 64 units to the model:
    model.add(layers.Dense(64, activation="relu", input_shape=(32,)))
    # Add another:
    model.add(layers.Dense(64, activation="relu"))
    # Add a softmax layer with 10 output units:
    model.add(layers.Dense(10, activation="softmax"))

    model.compile(
        optimizer=tf.train.RMSPropOptimizer(0.01),
        loss=tf.keras.losses.categorical_crossentropy,
        metrics=[tf.keras.metrics.categorical_accuracy])
    return model


def random_one_hot_labels(shape):
    n, n_class = shape
    classes = np.random.randint(0, n_class, n)
    labels = np.zeros((n, n_class))
    labels[np.arange(n), classes] = 1
    return labels


ray.init()


@ray.remote
class Network(object):
    def __init__(self):
        self.model = create_keras_model()
        self.dataset = np.random.random((1000, 32))
        self.labels = random_one_hot_labels((1000, 10))

    def train(self):
        history = self.model.fit(self.dataset, self.labels, verbose=False)
        return history.history

    def get_weights(self):
        return self.model.get_weights()

    def set_weights(self, weights):
        # Note that for simplicity this does not handle the optimizer state.
        self.model.set_weights(weights)


NetworkActor = Network.remote()
result_object_id = NetworkActor.train.remote()
ray.get(result_object_id)

0reactions

stale[bot]commented, Dec 7, 2020

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you’d still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray’s public slack channel.

Thanks again for opening the issue!

Top Results From Across the Web

Error when running code using pickle load - python

Never mind, I solved it, the problem is on the python version that doesn't support TensorFlow version 2.0.0, I guess pickle can work...

Using the SavedModel format | TensorFlow Core

When loading from a pickle file, if no variable_name is specified in the square brackets, whatever that is inside the pickle file will...

How to correctly install Keras and Tensorflow - ActiveState

Click to install Keras and Tensorflow together using pip. Understand how to use these Python libraries for machine learning use cases.

Is it possible to Play tf2 on a pickle powered computer ?(literally)

If you get an error message when launching Team Fortress 2, ... On sv_pure 0 servers, this mod will work perfectly, but there's...

Steam Workshop::[ArcStims] Pickle Stim (TTT/2 & Sandbox)

In sandbox, you can roll around and yell "I'M PICKLE RICK" and burp to ... Remember to download the prop in the Required...

Troubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.

Start Free

Top Related Reddit Thread

No results found

Top Related Tweet

No results found

Top Related Dev.to Post

No results found

Pickle Error with TF2.0

System information

Describe the problem

Source code / logs

Issue Analytics

Top GitHub Comments

Top Results From Across the Web

Top Related Medium Post

Top Related StackOverflow Question

Troubleshoot Live Code

Top Related Reddit Thread

Top Related Hackernoon Post

Top Related Tweet

Top Related Dev.to Post

Top Related Hashnode Post

Ray is not finding GPU but TF, PyTorch and nvcc does

MADDPG with horizon