Rethink the Third Party Modules registry

This issue has already been brought up multiple times in the old repo #26, #117 and in this issue comment on the main repo. In the meantime this new repo came up, and the old issues were archived. Now with v1 dropping and users starting to adopt Deno, this becomes more and more important, which is why I want to open an issue for it.

The Problem

Modules can be added to the database.json to make them show up on the Third Party Modules page https://deno.land/x/ and available under the URL https://deno.land/x/MODULE_NAME@BRANCH/SCRIPT.ts.

There are a few reasons why this is a suboptimal idea.

1. Scale

If Deno becomes even only half as popular as Node, there would be hundreds of modules added each day. It will totally flood the commit history and make the repo hard to maintain. Every PR needs to be verified by a maintainer, otherwise people could manipulate entries in any way. This is bad. Just looking at the recent commit history most of the commits are already related to adding entries for Third Party Modules.

1. Unique module names

Since module names must be unique, this will give rise to bad naming conflicts. Only the first module can be added, leaving anybody else with the same module name left out. This will be more like a domain name registry where all the good names are already taken. Even worse, entries for old unmaintained modules will block forever, because somebody out there might still depend on the URL so they can’t be removed. This is even worse than domain names, which can at least expire.

1. GitHub only

The current system allows only links to GitHub registries. This might currently not be a problem, since GitHub is the defacto standard Git repository host, but inherently being locked into one host by choice of a too narrow system is never good. Also limiting the registry to GitHub is effectively centralizing it, which achieves the opposite effect of what Deno tries to achieve by using the ES module system.

In summary, the current module registry doesn’t scale. It is high maintanance, requires unique module names, and is centralised.

A solution

What people need is an overview of modules compatible with Deno and a convenient way to link to them.

Currently the Third Party Modules registry solves both, albeit in a very limited way as highlighted above. Entries in the database.json are used to build the overview, as well as generate the links available through https://deno.land/x/....

I believe these two features shouldn’t be made dependent on each other, but looked at individually.

• Module overview

There needs to be a way for developers to add (and remove) modules that scales well for lots of modules each day. Also it shouldn’t be to the burden of the Deno maintainers, there should be an automated system of some sorts.

One idea would be to generate the overview dynamically by some form of a search algorithm. Essentially a Google for Deno modules. For example, a table with entries of all repositories that are tagged “deno” (https://github.com/topics/deno). The single source of truth is the repository and the generated table would adapt dynamically to any changes of the repo, like name, description, or even deletion. Repositories wouldn’t need to duplicate the name and description in another place and keep maintaining it if something changes. This would scale much more nicely, as no manual work is needed anymore and Deno maintainers could get back to writing code instead of verifying PRs. Repositories just need to add the tag “deno” to publish their modules which most already have anyways.

There are still some drawbacks to this that should be mentioned. There will be false positive matches by repositories adding the search tag for an unrelated reason. This could be mitigated by a sufficiently specific tag, but never fully solved. Also it doesn’t solve the centralization problem, as other hosting providers could be added to the search algorithm, but only so many. Modules hosted on a private website would be hard to add to the search algorithm.

• Module URL

If the module overview from above had a convenient way to obtain the URL of a module, then there would be little use case for an URL shortener like https://deno.land/x/.... For example, there could be a “Copy URL” button in the dynamically generated table which directly copies the github.com URL such that it can be pasted into the code, similar to what Google Fonts does.

Direct URLs also solve the issue of centralization. Obtaining all modules through the same domain makes the registry fragile. What if https://deno.land/x/... experienced outages, or even attacks? Also the URL shortener goes against the idea of the decentralization that Deno tries to achieve with the ES module system. Making a whole ecosystem depend on the uptime of a single domain is not good. Only raw direct links to wherever the module is hosted are truly decentralized, with no dependency on a single URL proxy.

Btw. GitHub URLs are actually pretty readable already making another point for abandoning the URL shortener https://deno.land/x/....

https://raw.githubusercontent.com/USERNAME/MODULE_NAME/BRANCH/SCRIPT.ts

Demo

I made a simple demo of repositories tagged with “deno” on GitHub which however still has the shortcomings mentioned above.

There are several false positives, like personal projects, editor extensions, deno itself, etc.

The mod.ts link serves as a standin for the “Copy URL” button. It also shows another problem of the search algorithm approach, as there is no predictable way to link to the entry script. Relying on a convention would not work, and the link would need to point to the repo where people need to read the actual documentation to find out what to import.

Also the console throws an error after the 10th page, since GitHub API limits unauthenticated requests to 10 per minute. Sicne each page only loads 30 items, the script would need a GitHub authentication to show more than 300 results.

TLDR

The Third Party Modules page https://deno.land/x/ could generate an overview of compatible modules with Deno dynamically from the “deno” tag on GitHub, to not depend on manual addition of modules to database.json. The URL service could be expanded to work as a full URL alias or just be abandoned altogether in favor of convenient links to GitHub directly from the table.

Summary

If Deno grows just as huge as we all hope, then the current Third Party Module registry page just won’t do. We should fix it rather soon with the few hundred modules, while we still got a chance.

EDIT: When I started this issue I favored the dynamic search approach. However by now I believe an actual registry would be better, due to decentrailzation not being limited by the search algorithm. The registry should directly link to the modules, wherever they are hosted, be it on GitHub or a tiny personal website (not like NPM, which actually hosts the modules). If the registry makes it easy to obtain the URL, imagine Google Fonts, then there is little reason for using a centralized URL shortener.

EDIT: There is a great proposal by @hacked for a registry with an example implementation.

[ghost]

I’ve been thinking about this for quite some time now and came to the conclusion that it’s pretty much impossible to satisfy everyone with a single solution. I think it’s important to remember what Deno is trying to achieve at the core, which is discard the need for something like npm.

That does however not elminate the need for a registry, which allows searching and browsing modules. For that reason I’ve been working on a service for some time now, trying to incorporate the idea of being able to host a module wherever you like and at the same time provide a consistent way of importing those modules in your project.

If you’re interested, have a look at https://deno.to. For now, there’s only a landing page with an explanation of what the service might look like in the future.

To sum up the concept, you can publish your module on GitHub, GitLab or BitBucket and add it to the website, manually. It’s more or less a vanity URL service, similar to https://deno.land/x, but improved. All it does is redirect to the URL you provided.

It also addresses most of the issues mentioned above, such as

• Name Squatting - Duplicate module names are not a problem, as long as the namespace (the username of the account) is unique.

• Scalability - We won’t depend on a single database.json file, every user can manage their own modules.

• GitHub Only - For now, the idea is to limit it to GitHub, GitLab and BitBucket. Theoretically, it is possible to virtually allow any website, as long as a certain URL schema is satisfied.

• Consistent Imports - All modules can be imported with the same URL, keeping it consistent and easy to read, https://deno.to/<name>/<module>/<version>

• Decentralized - The idea of not hosting modules in a central place remains. You are free to host your module wherever you like.

• Searching - You would be able to browse and search through all modules that are available on the website.

The only disadvantage is the need for an account system, but it won’t be possible to have a service like this without it.

I’m still working on getting an alpha version up and running, but my time is somewhat limited at the moment. The consideration could be made to just open source it and let the community build it as a whole.

If there are better domain name suggestions, feel free to let me know and I’ll buy them if needed. I figured deno.to was clever because it redirects to somewhere.

I’m definitely going to keep working on it, the nice thing about Deno is that we don’t depend on just one solution. If this turns out to not be the way to go, it can always be discarded.

Feel free to share suggestions, improvements or criticism.

[axetroy]

The problem is similar to cybersquatting

As long as a good module name is registered, others will no longer be able to submit the module with the name

Even if this module has been deprecated and will not be updated for several years, it cannot be deleted

@denoland developers should pay more attention to the development of core modules and standard libraries. Instead of wasting a lot of time processing database.json 's Pull Request