[Advice/Seeking Help] Very Low LRAP Score

Hi,

I’m currently training the multi-label classifier using XL-Net, to classify questions -> to concepts (for an online-learning platform). There’s ~1.7k concepts and 15k+ questions.

here’s the model: model = MultiLabelClassificationModel('xlnet', 'xlnet-base-cased', num_labels=len(lo_id_cols), args={'train_batch_size':8, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 3, 'max_seq_length': 512, "fp16": False, 'overwrite_output_dir': True})

This is my final output: {'LRAP': 0.004577569729953515, 'eval_loss': 0.27328102769863666}

I’m not sure what i’m doing wrong to have such a low LRAP. Would it be possible to have the model train to improve the LRAP ? any help/advice would be appreciated.

[ThilinaRajapakse]

Keep in mind that Transformer models are not good at extreme multiclass/multilabel classification. Even with weights, 1700 classes might just be too many. You can try to model the problem as a hierarchical classification to circumvent this problem. For example, starting with 10 broad categories which further divide into subcategories. This too might not be straightforward in multilabel classification.

There is a new paper by Amazon that seems to tackle the problem but I haven’t looked at it properly yet.

[AbinayaM02]

Calculate the number of datapoints tagged as 1 for each class individually and calculate the proportion for each class. For example, if there are 4 classes with [100, 500, 1000, 10] datapoints each, then class weights will be [10, 2, 1, 100]. In your case, if a single question is tagged with n classes, it will get accounted in each of the n classes individaully.