Preloading : ClassificationModel always return the same output per class initialization ( for each subsequent predict() )

Describe the bug

From my experiments with Bert and DistillBert (I could not test others yet), if I load a model with :

model = ClassificationModel(model_type, model_path, use_cuda=False)

And do a prediction with model.predict([text])

I get a correct return on the first predict.

Now, if I do a new prediction, model.predict([new_text]), i always get the same result as the first one.

To Reproduce Steps to reproduce the behavior:

You can use this code from my use case to reproduce : (a service listening on port 8080)

from simpletransformers.classification import ClassificationModel
from simpletransformers.experimental.classification import ClassificationModel as ExperimentalClassificationModel

import json

from bottle import run, post, request, response

loaded_models = { }

@post('/process')
def process():
    msg = json.loads(request.body.read())
    model_name = msg['model_name']
    text = msg['text']
    model_type= msg['model_type']
    model_path= msg['model_path']
    experimental = msg['experimental']
    cuda= msg['cuda']

    if model_name not in loaded_models:
        if experimental:
            model =ExperimentalClassificationModel(model_type, model_path, use_cuda=cuda, sliding_window=True)
        else:
            model = ClassificationModel(model_type, model_path, use_cuda=cuda)
        loaded_models[model_name] = model
    else:
        model = loaded_models[model_name]

    prediction, output = model.predict([text])
    print(prediction)
    print(output)
    return {} # you can improve the return value by parsing the ndarray to dict

print("Running prediction models pre-loader")

run(host='0.0.0.0', port=8080, debug=True)

And another little script to call the model (will trigger the prediction which should be outputed to the console on the other script)

import http.client

preloaded_models = http.client.HTTPConnection('localhost', 8080)

text = "your text here"
model_name= 1 #a unique model name 
path = "/your/model/location"

req = { 'model_name' : model_name, "text" :text, 'model_type' : 'bert', 'cuda' : False, "model_path": path, "experimental" : False}

req = json.dumps(req)
preloaded_models.request('POST', '/process', req)

Expected behavior

I should get the correct model_output corresponding to the input text. This problem prevents pre-loading the models in memory for faster predictions.

#edit: typos

[TheCodingLand]

Thank you @ThilinaRajapakse , I managed to fix it. I added the option ‘reprocess_input_data’ to True, but it didn’t fix it, then I added ‘use_cached_eval_features’ : False. it now works.

from simpletransformers.classification import ClassificationModel
from simpletransformers.experimental.classification import ClassificationModel as ExperimentalClassificationModel

import json

from bottle import run, post, request, response

loaded_models = { }

@post('/process')
def process():
    msg = json.loads(request.body.read())
    model_name = msg['model_name']
    text = msg['text']
    model_type= msg['model_type']
    model_path= msg['model_path']
    experimental = msg['experimental']
    cuda= msg['cuda']

    if model_name not in loaded_models:
        if experimental:
            model =ExperimentalClassificationModel(model_type, model_path, use_cuda=cuda, sliding_window=True, args={'use_cached_eval_features' : False,'reprocess_input_data': True})
        else:
            model = ClassificationModel(model_type, model_path, use_cuda=cuda, args={'use_cached_eval_features' : False,'reprocess_input_data': True})
        loaded_models[model_name] = model
    else:
        model = loaded_models[model_name]

    prediction, output = model.predict([text])
    print(prediction)
    print(output)
    return {} # you can improve the return value by parsing the ndarray to dict

print("Running prediction models pre-loader")

run(host='0.0.0.0', port=8080, debug=True)

Great work on the library. I think just updating the documentation should be enough. It makes sense that we should specify this only for predictions, as they do not affect the training.

Maybe you could specify if arguments are used while training, predicting or evaluating the models in the arguments part of the documentation ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.