question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

function cannot be pickled for mongodb

See original GitHub issue

When using hyperopt.mongoexp.MongoTrials for parallelism, I am getting the same error as expressed in this StackOverflow thread. In my case, because I am using hyperas, the error says hyperopt.mongoexp:job exception: No module named temp_model.

I believe I have made my function self-sufficient, and I have installed dill. Is there anything done to the function by hyperas that would make it temp_function impossible to pickle? Has anyone else successfully used hyperas with mongodb?

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Reactions:1
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

3reactions
JonnoFTWcommented, Mar 17, 2018

I got it to work by commenting out the lines in optim.py that removed temp_model.py. From there I:

  • specified an absolute path to my data file in data()
  • Set my trials to: trials = mongoexp.MongoTrials('mongo://localhost:27017/jobs/jobs', exp_key='htm_113_108')
  • set $PYTHONPATH to include my project directory containing temp_model.py
  • Ran the worker script with: ./hyperopt-mongo-worker --mongo=localhost:27017/jobs --exp-key=htm_113_108
1reaction
maxpumperlacommented, Nov 7, 2017

@mcgibbon @myhussien sorry, this one slipped my attention. I’ve never used hyperas with mongodb, but it’s doomed to fail. hyperas writes a temp file that the mongodb workers have to have access to. In principle that can work, but hyperas (as of now) isn’t built for it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python multiprocessing, can't pickle thread.lock (pymongo. ...
I am using Mongodb 5.0 and pymongo client. ... TypeError: cannot pickle '_thread.lock' object Exception ignored in: <function CommandCursor.
Read more >
[PYTHON-2415] Can't pickle BulkWriteError
Can't pickle BulkWriteError ; Component/s: Error Handling ; Labels: size-small ; Environment: CPython 3.8.5 ; import pickle ; import pymongo.
Read more >
Python 3 FAQ — PyMongo 3.2.2 documentation
¶ Instances of ObjectId pickled using Python 2 can always be unpickled using Python 3. Due to http://bugs.python.org/issue13505 you must use Python 3.2....
Read more >
cannot pickle 'SSLContext' object-mongodb
[Solved]-Mongodb aggregation in Python : cannot pickle 'SSLContext' object-mongodb ... It's almost certainly not your query per se; the issue will be with...
Read more >
Can't pickle functions & code objects problem - Scripting
Hi, Python node has the option to pass data formats other than table with object port, however it cannot pass neither a function...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found