question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Automatic model download doesn't work

See original GitHub issue

Issue Description

mp_rank_00_model_states.pt ends up containing:

<!DOCTYPE html><html><head><title>Google Drive - Virus scan warning</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><style nonce="t256RPQHLynZvvCq0ggl7w">/* Copyright 2022 Google Inc. All Rights Reserved. */
.goog-inline-block{position:relative;display:-moz-inline-box;display:inline-block}* html .goog-inline-block,*:first-child+html .goog-inline-block{display:inline}.goog-link-button{position:relative;color:#15c;text-decoration:underline;cursor:pointer}.goog-link-button-disabled{color:#ccc;text-decoration:none;cursor:default}body{color:#222;font:normal 13px/1.4 arial,sans-serif;margin:0}.grecaptcha-badge{visibility:hidden}.uc-main{padding-top:50px;text-align:center}#uc-dl-icon{display:inline-block;margin-top:16px;padding-right:1em;vertical-align:top}#uc-text{display:inline-block;max-width:68ex;text-align:left}.uc-error-caption,.uc-warning-caption{color:#222;font-size:16px}#uc-download-link{text-decoration:none}.uc-name-size a{color:#15c;text-decoration:none}.uc-name-size a:visited{color:#61c;text-decoration:none}.uc-name-size a:active{color:#d14836;text-decoration:none}.uc-footer{color:#777;font-size:11px;padding-bottom:5ex;padding-top:5ex;text-align:center}.uc-footer a{color:#15c}.uc-footer a:visited{color:#61c}.uc-footer a:active{color:#d14836}.uc-footer-divider{color:#ccc;width:100%}</style><link rel="icon" href="null"/></head><body><div class="uc-main"><div id="uc-dl-icon" class="image-container"><div class="drive-sprite-aux-download-file"></div></div><div id="uc-text"><p class="uc-warning-caption">Google Drive can't scan this file for viruses.</p><p class="uc-warning-subcaption"><span class="uc-name-size"><a href="/open?id=1EiAY3IcKWmGADaLDzdG25ykQghUwza6L">mp_rank_00_model_states.pt</a> (12G)</span> is too large for Google to scan for viruses. Would you still like to download this file?</p><form id="downloadForm" action="https://drive.google.com/uc?id=1EiAY3IcKWmGADaLDzdG25ykQghUwza6L&amp;export=download&amp;confirm=t" method="post"><input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/></form></div></div><div class="uc-footer"><hr class="uc-footer-divider"></div></body></html>

causing:

Traceback (most recent call last):
  File "/home/ubuntu/magma/example_inference.py", line 4, in <module>
    model = Magma.from_checkpoint(
  File "/home/ubuntu/magma/magma/magma.py", line 292, in from_checkpoint
    sd = torch.load(checkpoint_path, map_location=torch.device("cpu"))
  File "/usr/local/share/miniconda/lib/python3.9/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/share/miniconda/lib/python3.9/site-packages/torch/serialization.py", line 762, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'.

Possibly related to https://github.com/wkentaro/gdown/issues/26

Issue Analytics

  • State:closed
  • Created 10 months ago
  • Reactions:1
  • Comments:5

github_iconTop GitHub Comments

4reactions
stefan-itcommented, Mar 15, 2022

Hey guys,

I think another good option would be to also host the model(s) on Hugging Face model hub.

They also provide huggingface_hub library that can be used as a nice client to get the model download integrated into magma.

We have been using this in Flair library and also hosting some of our models on the model hub.

3reactions
osansevierocommented, Mar 17, 2022

Hey all, I’m Omar from HF 🤗

We would love to support having your models in the Hub. One thing that you might find useful is that model cards have metadata that allow reporting things such as the dataset, metrics, tags, etc. This can help with discoverability, which would be much better than through Google Drive links.

The model could live within an organization for Aleph Alpha for additional visibility, and we have a lightweight Python library called huggingface_hub that would allow users to download/cache the model locally. Would you be interested in this?

Adding the model is relatively simple, here are some instructions for it -> https://huggingface.co/docs/hub/adding-a-model. I actually have one copy here which we could transfer to your org and add a nice model card.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Sharpen 4.0 or 4.1 Model Download Is Slow Or Not Working
Sharpen 4.0 or 4.1 Model Download Is Slow Or Not Working · 1) Close the app and download the models folder from: https://drive.google.com/file/d/ ......
Read more >
Fix file download errors - Google Chrome Help
If you try to download a file and it doesn't work, try to fix the error with these troubleshooting steps: ... Chrome will...
Read more >
SFM will not download/update any models - Steam Community
Open your Library in Steam and right click on Source Filmmaker, select 'Properties' then select 'Set Launch Options' and make sure that "- ......
Read more >
Downloading 3D models - Sketchfab Help Center
If the Download 3D Model link is not visible on the model page (like the screenshot above), the user has decided not to...
Read more >
Sketchfab: Sometimes It Works . Sometimes It Doesn't. Any ...
Please try to recreate this issue, opening Twinmotion and downloading Sketchfab assets until you aren't able to download assets anymore. As soon ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found