Using a specific transformer encoder model for Text Classification Task

Hi, I would like to use a specific transformer encoder model such as roberta-large instead of the default roberta, which loads a roberta-base. Is there any way to do so?

[justinxzhao]

Hi,

You can use auto_transformer, i.e.:

import pandas as pd
import yaml

from ludwig.api import LudwigModel

config = """
input_features:
    - name: text
      type: text
      encoder: auto_transformer
      pretrained_model_name_or_path: 'roberta-large'
output_features:
    - name: category
      type: category
trainer:
    epochs: 1
"""
model = LudwigModel(yaml.load(config), backend="local")

df = pd.DataFrame(
    {
        "text": ["Suomessa vaihtuu kesän aikana sekä pääministeri että valtiovarain"],
        "category": ["Suomi"],
    }
)
model.train(df)
model.predict(df)

Related discussion: https://github.com/ludwig-ai/ludwig/discussions/2057

Documentation: https://ludwig-ai.github.io/ludwig-docs/0.5/configuration/features/text_features/#autotransformer

[msakthiganesh]

Hi @justinxzhao ! I recently came across a similar machine learning framework - Lightwood .

On comparison with Ludwig, they both have functionalities to handle features of multiple input types, pre-process them based on their input type, encode the inputs, TRAIN THE MODEL USING THE ENCODED INPUT (emphasizing on the model), decode the outputs from the trained model into output features.

In Lightwood, the possible models that could be trained are provided here - https://lightwood.io/mixer.html

In similar fashion, are there a list of possible models that could be trained using Ludwig? And if yes, how do we specify our preferred model to train (using the concated output from the combiner)?

Thanks!