TypeError: 'generator' object is not callable

I’m getting the exception TypeError: ‘generator’ object is not callable when I train with multiple GPU’s

I’m not sure where it’s coming from, my datasets are subclasses of torchtext.data.Dataset and the data loaders are torchtext.data.BucketIterator.

What’s the easiest way of identifying what’s causing the exception?

    # create trainer
    mlflow = loggers.MLFlowLogger("Transformer")
    trainer = pl.Trainer.from_argparse_args(args, early_stop_callback=True, min_epochs=10, logger=mlflow, gpus=[0,1])

    # prepare data
    train_dataset, val_dataset = load_datasets(path=args.path, files=args.files)
    train_loader, val_loader = create_dataloaders(train_dataset, val_dataset, batch_size=args.batch_size)

    # init the model
    hparams = vars(args)
    transformer = Transformer(src_vocab=train_dataset.vocab, tgt_vocab=val_dataset.vocab, **hparams)

    # train
    trainer.fit(transformer, train_loader, val_dataloaders=val_loader)
    transformer.freeze()

[edenlightning]

@awaelchli mind taking a look?

[david-waterworth]

@awaelchli thanks for looking. I need to refactor this anyway since as I mentioned torchtext has started to deprecate their DataLoader / Dataset in place of the torch equivalents. I’ve been putting it off since they’ve not yet provided a migration path for custom datasets (frustratingly they seem to be focusing on migrating the built-in datasets using private pipeline methods - I feel they should have created a migration pathway first and then used it to migrate their datasets).

@mateuszpieniak any pointers would be useful - I need to add ‘<sos>’ and ‘<eos>’ tokens to each example and then pad each batch. Ideally, by using high-performance code someone has already thoroughly tested