Data preprocessing for 20newsgroups dataset

Description

Hi, guys. First of all, thanks for sharing your work with us. I’d like to reproduce the experiments on CTM paper using the 20newsgroups dataset. I have some questions:

• are there any additional preprocessing steps other than those reported on paper (“removing digits, punctuation, stopwords, and infrequent words”), e.g. removal of email headers, email addresses etc?
• is there any data partition (e.g. train/validation), or the reported metrics are calculated on full training data?
• where does the count of 18,173 documents come from? I checked data from both sklearn and the url in the paper and they contain 18,846 docs.

Thanks in advance!

[A11en0]

Thanks a lot for your quick response! But if there is no split, would the model overfit the datasets?

Yes, it’s a little bit strange. A topic model is different from the normal cluster task. It can be made an inference on the unseen dataset. I read some papers that split the evaluate type with transductive and inductive, the former suggests using the full dataset, and later like a classification task i.e. train a model first on training sets and then inference on unseen testing sets. Refer to https://ojs.aaai.org/index.php/AAAI/article/view/6152.

[vinid]

Hello @marcospiau!

1. if you are trying to replicate our results, no, you can just apply those steps. We used the following setup to remove text that was not useful:

data = fetch_20newsgroups(subset='all',
                                       remove=('headers', 'footers', 'quotes'),
                                       categories=categories)

When we remove infrequent words we already remove most of the things like urls/adresses.

You can get the data we used here, might be easier for you just to use this if you need to reproduce:

https://raw.githubusercontent.com/silviatti/preprocessed_datasets/master/ctm/20news_unprep.txt https://raw.githubusercontent.com/silviatti/preprocessed_datasets/master/ctm/20news_prep.txt

1. reported metrics are computed on the topic lists, so there’s currently no validation involved. (we support early stopping but that functionality was not implemented at the time of the paper).

2. Some documents have to be removed after pre-processing because they became empty

Let me know if you have more questions 😃