Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Exposing all of npm through dat to dep
See original GitHub issueHi @watilde. Dep made me do a new version of an old project I never finished. I have pushed new commits to https://github.com/mafintosh/dat-npm and am running it on a server.
The goal is to have a single Dat with all of NPM in it. There are two steps. Step 1 is to collect all of the NPM registry metadata into a single hyperdb (hypercore). We expose it as a hyperdb like this: https://github.com/mafintosh/dat-npm/blob/master/get.js (note this is broken because of a hyperdb bug but will be fixed soon). The data returned looks like this:
{
"0.0.1": {
"dependencies": {
"xml2js-expat": "0.2.0"
},
"devDependencies": {}
},
"0.1.0": {
"dependencies": {
"xml2js-expat": "0.2.0"
},
"devDependencies": {}
},
"0.2.2": {
"dependencies": {
"xml2js-expat": "0.2.0"
},
"optionalDependencies": {},
"devDependencies": {
"mocha": "0.x.x"
}
},
"0.2.3": {
"dependencies": {
"xml2js-expat": "0.2.x"
},
"devDependencies": {
"mocha": "0.x.x"
}
}
}
Step 2 is to distribute the NPM tarballs over Dat. Because the tarballs are very large, we do not want to download them all up front. We want to download them on-demand. Dat does not have a mechanism yet for requesting files on-demand, so I was thinking we could have a simple REST API where you could do GET https://npm.datproject.org/request@1.2.1 and the response would return 200 OK when the request@1.2.1 tarball has been added to the npm-dat. Then the client program could request the tarball from the Dat repository and it would be there.
To integrate with dep I have some questions:
- How would a user tell
depto use Dat for modules instead of npm? Maybe--datflag in the CLI and it would ue Dat for everything? Or adat: truesetting in a config somewhere? A user could also specify it one at a time for each dependency, but I think it would be cool to telldepto use Dat for all modules also. - Is the version metadata above enough for
depto do recursive module resolution, or do you need other info from package.json? - In the
depcode, how much work would it be to integrate the Step 1 hyperdb as a registry for metadata? - In the
depcode, how much work would it be to integrate the Step 2 Dat as the source of tarballs?
For the dep integration I imagine an algorithm like this:
- User types
dep install --dat depreads package.json, doeshyperdb.get('/modules/<pkg>', cb)for each onedeprecursively resolves metadata usinghyperdb- Each time
depneeds a tarball, it wouldGET https://npm.datproject.org/module@version - When response comes back 200 OK
depwould then read the tarball from the Dat
In the future we can replace the npm.datproject.org server with a pure-Dat solution, but for now the REST API is a way we can deploy this sooner.
Issue Analytics
- State:
- Created 6 years ago
- Reactions:9
- Comments:19 (3 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
OK it’s all imported and live updating from the changes feed:
@watilde I accidentally crashed NPM yesterday (sorry NPM) 😂
I changed my approach. Now I am downloading all tarballs as I process the NPM changes feed. This is the way to integrate it now: https://github.com/mafintosh/dat-npm/blob/master/get.js
The changes feed processor will take a while to finish, so not all modules will show up in the query. For each change, it downloads all tarballs and writes them to the tarballs hyperdrive, then writes the metadata to the metadata hyperdb. So if the metadata record is there, then all the tarballs will be ready. The example
get.jsuses the ‘pushpop’ module which has already been written. Here’s what the server logs look like:I’ll report back when the import is finished. @watilde is this enough for you to integrate into
dep?