RFC Split the repo based on the projects in the repo

Right now the huggingface_repo includes three projects;

• huggingface_hub: which is the core library used by integration libraries and api-inference-community
• api-inference-community which handes integration with some of the third party libraries
• huggingface_widgets which is located under the js folder and hosts the code for the inference widgets

The three projects require different dev environments, have different development styles, and follow different release cycles.

This request for comment is for us to see if it would make sense to separate the repo into three different repositories.

cc @julien-c @LysandreJik @osanseviero @Narsil

[pngwn]

I personally have no opinion on this suggestion, as I do not routinely work with these projects but thought I would share my experience.

In my experience, lots of individual repos scales poorly as an org grows. Having fewer, bigger repos, increases the likelihood that the repo will be well maintained (as it is less liklely to get abandoned without intention, and generally has more eyes on it), offers an easier path for code sharing (local files/ local symlinks are far easier to maintain and utilise than cross-repository ones), makes enforcing code and quality standards easier and, in particular, makes sharing CI easier.

I don’t think there are any inherent problems with multi-language mono-repos, whether custom tooling is used to support that or not.

I also think there are better mechanisms for importing dependencies than relying on the entire repo. Package repositories, both public and private, exist to address that problem.

[stevhliu]

+1 for the split! ❤️

Not directly related to splitting the repo, but I think this could be a good opportunity to sort of refresh the Hub product docs. Instead of a list of questions (i.e, How can I do this? Can I do that?), I think it would be nice to give it a little more narrative/structure and focus more on user goals instead of more granular/narrow questions.