ran out of memory

Hey guys,

I’m trying to use fairseq to train a simple summarizer like a translation task. oom happened where even though when I set --max-tokens to 500. 100 or smaller max-tokens set got error:

Traceback (most recent call last):
  File "train.py", line 359, in <module>
    main(args)
  File "train.py", line 78, in main
    trainer.dummy_train_step([dummy_batch])
  File "/home/fairseq/fairseq/trainer.py", line 326, in dummy_train_step
    self.train_step(dummy_batch, dummy_batch=True)
  File "/home/fairseq/fairseq/trainer.py", line 176, in train_step
    self.model, self.criterion, sample,
  File "/home/fairseq/fairseq/tasks/fairseq_task.py", line 157, in get_loss
    return criterion(model, sample)
  File "/home/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/fairseq/fairseq/criterions/cross_entropy.py", line 30, in forward
    net_output = model(**sample['net_input'])
TypeError: 'NoneType' object is not subscriptable

So, any suggestion? My dataset contains about 100,000 sentences and my gpu is single 1080ti 11G memory.

[edunov]

Another thing to try is to apply BPE encoding to your data, that will naturally reduce dictionary size. There are examples in our repository: https://github.com/pytorch/fairseq/blob/master/examples/translation/prepare-wmt14en2de.sh#L125-L132

[huihuifan]

1 million is too large. try --thresholdsrc --thresholdtgt