Distributed TFIDF

Greetings!

I recently used dask to implement a distributed version of tfidf. I want to contribute to the dask project by putting it somewhere.

Would this be the correct repo.?

I thought maybe a feature_extraction directory would be appropriate.

[TomAugspurger]

What’s the difference between this repo. and dask-glm?

I’m probably going to just import the dask-glm estimators into dask-ml namespace (likewise with dask-searchcv, dask-patternsearch). For the user, it’d be nice to have a single place to go for all dask-related ML things.

Development will probably still continue in those other repositories.

[mrocklin]

+1 on avoiding bag in performance sensitive code 😃

On Wed, Jan 24, 2018 at 5:56 PM, Roman Yurchak [e-mail] wrote:

An alternative approach to using dask bag could be to apply scikit-learn CountVectorizer or HashingVectorizer on chunks of the dataset, merge the results and then apply IDF weighting (see FreeDiscovery/FreeDiscovery#152 https://github.com/FreeDiscovery/FreeDiscovery/issues/152). It might require somewhat less work since different vectorization options are already implemented in scikit-learn, and it should be possible to keep a fairly compatible API.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/dask-ml/issues/5#issuecomment-360302286, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszBORFY_eHqD6DQEuYQgD4cNmXcsYks5tN7UigaJpZM4PiBOZ .