(kMeans, cosine) wrong number of centroids (?)

I’m trying to cluster sentences using TF-IDF, cosine similarity and k-means algorithm. I calculate TF-IDF using natural library then I prepare vectors that look like:

[0, 0, 0, 0, 0, 1.24225242, 0, 0, 0, 0, 0] where every value represents a word and the whole vector represents a sentence (in case of this example it would be a sentence with one word).

Then I try to use talisman library to make clusters using created vectors and metrics/distance/cosine. The problem is that I get:

TypeError: Cannot read property 'length' of undefined
    at KMeans.cosine [as distance] (.../node_modules/talisman/metrics/distance/cosine.js:29:21)
    at KMeans.iterate (.../node_modules/talisman/clustering/k-means.js:164:22)
    at KMeans.run (.../node_modules/talisman/clustering/k-means.js:224:12)
    at kMeans (.../node_modules/talisman/clustering/k-means.js:243:21)
    at Object.<anonymous> (.../lev-kmeans.js:39:22)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

I tried looking into the code and the reason it fails is this piece of code (https://github.com/Yomguithereal/talisman/blob/master/src/clustering/k-means.js#L148).

for (let j = 0, m = this.dimensions; j < m; j++) {
     const d = this.distance(vector, this.centroids[j]);

     if (d < min) {
         min = d;
         minIndex = j;
     }
}

It tries to enter cosine.js functions using undefined as the second vector. For example for k equal 2 it should generate 2 centroids. It does. But this loop is trying to access centroids with very high indices like 181 (this.dimensions and thus m is equal 182).

It works fine when I tested it for simpler stuff using Euclidean distances.

To be complete I add the way I initialize the kMeans:

const clusters = kMeans({
    distance: cosine,
    k: 6,
}, vectors)

Where cosine is a const cosine = require('talisman/metrics/distance/cosine').

[Yomguithereal]

Ok, the thing was that in the data you provided in the repo, the description key seems to always yield only three tokens.

So are we going in the good direction?

[galkowskit]

Interesting. Do you mean to keep, for instance, the top n tokens of your document using TF-IDF score?

Yes. Basically each cluster would be summarized by top n TF-IDF tokens and those would be saved. Just for auto-tagging. I was thinking that those could also be useful for classification.