Possible to reuse features?

I have a binary classification task, that I want to tackle with the ClassificationModel and albert from this library.

In my case, the prediction results are not yet as great as I want them to be, so I want to label a lot more data with support of the predictions. Concretely, I want to calculate the confidence of a result as described here and then label the samples with low confidence.

I want to achieve that by just training n albert models, e.g. 10 of them, with the same data. Then I run the prediction with all of them and can see, for which predictions “they have the same opinion” (low variance), and where they have low confidence, so a high variance.

As of now, I’m naively training 10 ClassificationModels, save them and then loop through all of them with model.predict. Then I calculate the variance using the variance formula.

This works perfectly, however now I need 3s per data sample to calculate this.

What are some possible methods we can speed this up? If I understand correctly, the features can probably not be reused, as all albert layers and embeddings in the models are different after the training, as all weights in albert are affected by the training.

I was thinking about using ONNX to speed things up, but I’m not sure how much it’ll bring and how to transform a ClassificationModel to ONNX.

I know I pointed to multiple things here, so let me know if I should split stuff up into multiple issues 😃

[ThilinaRajapakse]

No problem!

Google Colab might be an option in that case.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.