question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

dvc push: it's run successfully but the remote path has no file

See original GitHub issue

Bug Report

Description

I setup a hdfs path as remote storage, and dvc add and then push a file successfully according to the terminal message. But I can’t see files in the hdfs path.

Reproduce

0, dvc init 1, dvc add data.csv 2, dvc remote add -d myhdfs hdfs://xxx/yyy 3, dvc push 4, hadoop fs -ls hdfs://xxx/yyy

Expected

Expect to see some files in the hdfs path after pushing successfully, but got nothing there.

Environment information

Output of hadoop fs -ls hdfs://xxx/yyy:

$ 

Additional Information (if any): After runing dvc push, the output is

$ 0% Querying cache in yyy|                                                                                                                                        
22/05/24 20:50:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/05/24 20:50:26 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Everything is up to date.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
zhihuacccommented, May 24, 2022

@daavoo I tried 2.10.2, and it worked. I can push to and find the files in my remote path.

0reactions
daavoocommented, May 24, 2022

@daavoo dvc version is 2.10.1

Could you try with 2.10.2? I think this was solved in https://github.com/iterative/dvc/pull/7563

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting | Data Version Control - DVC
The most common cause is changes pushed to Git without the corresponding data being uploaded to the DVC remote. Make sure to dvc...
Read more >
dvc push: when not all files are in local cache, still ... - GitHub
The issue was pushing incomplete datasets from my gpu server to the new storage remotes. It pushed all the files present in the...
Read more >
Data & Model Management with DVC | Analytics Vidhya
In this post we learn about versioning for ML projects & use DVC to version & maintain ML artifacts in a remote Amazon...
Read more >
Cannot add 'folder-path', because it is overlapping with other ...
dvc files in my project. $ dvc add ./project_model/data/ ERROR: Cannot add '/home/me/PycharmProjects/project/project_model/data/images' ...
Read more >
November '19 DVC❤️Heartbeat - Iterative.ai
Git commit id (hash) is saved in DVC file upon import, data itself is copied/downloaded into DVC repo cache but would not be...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found