A different way of doing the similarity/comparison task?

Hey! Thanks for the awsome work. I was wondering if I could use and update finetune to do the following:

Instead of using (Start, Text1, Delim, Text2, Extract) and (Start, Text2, Delim, Text1, Extract) as in the paper, can we use (Start, Text1, Extract) and (Start, Text2, Extract) separately through the transformer?

This could be thought of as obtaining sentence/document embeddings for Text1 and Text2 separately. Upon doing that, I would like to compare their similarity using a distance metric such as cosine distance. (i.e. train the transformer as a siamese network.)

Would you suggest I build such a model on top of a fork of finetune?

[madisonmay]

Hey @chaitjo, glad you’ve been hacking away over the weekend! I’ve opened up a PR into development for you – don’t worry if things don’t run yet I’ll just use it as a space to leave code comments for now.

At a high level things look good, seems like there are just a few things to clean up. Perhaps most importantly, I think we can find a way around overriding many of the BaseModel methods by structuring things more like the comparison class (and maybe inheriting from that instead of the BaseModel class.)

As far as data goes, a sampled version of the Quora similarity dataset might be a good place to start. We’ve got some scripts in there for training already, but note that you’ll probably have to modify those scripts to convert the pandas Series passed as inputs to numpy arrays – that’s on my backlog to patch up. E.g.:

    dataset = QuoraDuplicate(nrows=100).dataframe
    model = Comparison(verbose=True, n_epochs=3, lm_warmup=0.1)
    trainX1, testX1, trainX2, testX2, trainY, testY = train_test_split(dataset.Text1, dataset.Text2, dataset.Target, test_size=0.3, random_state=42)
    model.fit(list(zip(trainX1.values, trainX2.values)), trainY.values)
    accuracy = np.mean(model.predict(list(zip(testX1.values, testX2.values))) == testY.values)
    class_balance = np.mean(testY.values)
    print('Test Accuracy: {:0.2f} for a {:0.2f} class balance'.format(accuracy, class_balance))

[chaitjo]

Thanks for the response! Will keep you posted on the progress.