How to make TabularDataset loading csv faster

I try torchtext with data from https://www.kaggle.com/c/quora-question-pairs/data, which size is 200MB.

train.csv: 404302 lines test.csv:3563490 lines

I use TabularDataset to load them, it’s too slow. For train.csv, it cost 5mins, and for test.csv, it can’t finish in 40mins. But it only need 3sec when use pandas to load train.csv, and 20 sec for keras’s texts_to_sequences to process.

here is my code, does anything wrong: `print(“Preparing CSV files…”)

QUESTION = data.Field(
    sequential=True,
    fix_length=fix_length,
    tokenize=tokenizer,
    init_token='SOS',
    eos_token='EOS',
    lower=lower
)

LABEL = data.Field(
    sequential=False,
)

print("Reading train csv file...")
train = data.TabularDataset(
    path= mypath + '/train.csv', format='csv', skip_header=True,
    fields=[
        ('id', None),
        ('qid1', None),
        ('qid2', None),
        ('question1', QUESTION),
        ('question2', QUESTION),
        ('label', LABEL),
    ])

print(vars(train[0]))` 

[BartekRoszak]

What tokenizer do you use? Maybe it could be better if you pre-tokenize text and join tokens with whitespace. Then you will need only str.split as tokenizer which works fast.

[prashantbudania]

You should disable spacy’s pipelines if you are only using it for tokenization. More info here: https://spacy.io/usage/processing-pipelines#disabling

(because parser pipeline is the slowest and if you are not using that information, then it’s just waste of time)