Tokenizer's nb_words not taking effect when fit_on_texts is run

Hello, I can’t figure out if I’m doing something wrong, but Tokenizer always seems to ignore the nb_words parameter I provide it and tokenize ALL words rather than just the top nb_words. I’m running Python 2.7. Note that I ran into this issue while working on a dataset with over 1000 unique words, and any lower value I set for nb_words (e.g. 10, 100, 500…) was ignored. Below is a simple example to illustrate quickly what I’m getting. Thank you.

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(nb_words=10)
tokenizer.fit_on_texts(['apple book car dog egg fries girl ham inside jam knife leg monkey nod open pear question rough stone tree umbrella voice wax xylophone year zoo'])
print(len(tokenizer.word_index))
# comes out as 26 rather than 10

[shafy]

Is this behavior intended? If yes, it would be very useful to have a parameter to only get the word_index for the top nb_words

[hsiaoyi0504]

@shafy I mean you can pass the word to word_index and check if the mapping index is under the nb_words. It’s not a hard task. If the word_index dictionary is large enough, you may want generate a filtered version rather than doing this on the fly.