Evaluate 3 different topic modeling algorithms

• OCTIS version:
• Python version:3,7
• Operating System: linux

Description

I am a PhD candidate and I need to evaluate the performance of three different topic model algorithm including: LDA, LSI and Bertopic. ( LDA and LSI were trained using the Gensim package) what are the relevance metrics that I should use apart from coherence score? I would like to include in my paper a sort of table or graph that shows an evaluation in term of accuracy of the model (coherence score) and relevance of topics ( should I use the topic diversity metric ?) Thank you

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

[silviatti]

Which diversity metric are you using? Can you also show the snippet of the code in which you call the metric? In general a metric in OCTIS expects to receive in input the output of a Model(). Any topic model in OCTIS returns a dictionary with up to 4 fields. Depending on the metric, the right field will be used to compute the metric (see here for the details on model_output) So if you want to use a metric that uses the word-topic distribution to compute the diversity, then you will construct your model_output like this:

model_output = {"topic-word-matrix": topic_term_dist}

And then use it to compute the score of a metric. For example,

div = KLDivergence()
result = div.score(model_output)

Let me know if it works.

Silvia

[silviatti]

Hello, it depends on what your objective is. Any evaluation metric focuses on a specific aspect of a topic model. OCTIS includes different categories of evaluation metrics:

• topic coherence metrics (evaluating if the top-words of the topics make sense together)
• topic significance metrics that consider the document-topic and word-topic distributions to discover high-quality and junk topics. You can find the reference paper here.
• classification metrics (F1, accuracy, etc), which use the document-topic distributions as features to train a classifier. These metrics require that documents are labelled.
• diversity metrics, which consider the top-words or the word-topic distribution and compute the distance between a topic and the others.

I am not sure if BERTopic generates the document-topic and word-topic distributions (in that case, you will not be able to compute the topic significance metrics). Maybe you’d like to consider Contextualized Topic Models (CTM) which is a topic model that uses pre-trained contextualized representations (as BERTopic). CTM is part of OCTIS too.

Let me know if you have further questions,

Silvia