Wrong indent for "results" variable prevents data set processing of squad like json file

Describe the bug Just a wrong indent of the variable “results”. In the current version, there is one indent and so the results just get executed if the else block is true before. That’s why in my case the results variable is not defined and I get the following error:

Error message

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/farm/data_handler/data_silo.py in _load_data(self, train_dicts, dev_dicts, test_dicts)
    188             train_file = self.processor.data_dir / self.processor.train_filename
    189             logger.info("Loading train set from: {} ".format(train_file))
--> 190             self.data["train"], self.tensor_names = self._get_dataset(train_file)
    191 
    192         # dev data

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/farm/data_handler/data_silo.py in _get_dataset(self, filename, dicts)
    159 
    160             with tqdm(total=len(dicts), unit=' Dicts', desc="Preprocessing Dataset") as pbar:
--> 161                 for dataset, tensor_names in results:
    162                     datasets.append(dataset)
    163                     pbar.update(multiprocessing_chunk_size)

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py in next(self, timeout)
    746         if success:
    747             return value
--> 748         raise value
    749 
    750     __next__ = next                    # XXX

IndexError: Cannot choose from an empty sequence

Expected behavior sequence should not be empty

To Reproduce Just try

# and fine-tune it on your own custom dataset (should be in SQuAD like format)
train_data = "training_data"
reader.train(data_dir=train_data, train_filename="2020-02-09_answers.json", use_gpu=False, n_epochs=1)

System:

• OS: MacOS Mojave
• GPU/CPU: CPU
• FARM version: 0.4.1

FIX: Under data_handler > data_silo.py > DataSilo > _get_dataset

with ExitStack() as stack:
            if self.max_processes > 1:  # use multiprocessing only when max_processes > 1
                p = stack.enter_context(mp.Pool(processes=num_cpus_used))

                logger.info(
                    f"Got ya {num_cpus_used} parallel workers to convert {num_dicts} dictionaries "
                    f"to pytorch datasets (chunksize = {multiprocessing_chunk_size})..."
                )
                log_ascii_workers(num_cpus_used, logger)

                results = p.imap(
                    partial(self._dataset_from_chunk, processor=self.processor),
                    grouper(dicts, multiprocessing_chunk_size),
                    chunksize=1,
                )
            else:
                logger.info(
                    f"Multiprocessing disabled, using a single worker to convert {num_dicts}"
                    f"dictionaries to pytorch datasets."
                )
            #######################################################
            ####fix indent here ###############
            results = map(partial(self._dataset_from_chunk, processor=self.processor), grouper(dicts, num_dicts))
            #########################################
            datasets = []

            with tqdm(total=len(dicts), unit=' Dicts', desc="Preprocessing Dataset") as pbar:
                for dataset, tensor_names in results:
                    datasets.append(dataset)
                    pbar.update(multiprocessing_chunk_size)

            concat_datasets = ConcatDataset(datasets)
            return concat_datasets, tensor_names

I would have created a pull request but I’m not able to create a new branch. Hopefully this helps. Let me know if you have further questions.

[RobKnop]

@tanaysoni Should I go ahead and close the issue? That’s fine for me.

[tanaysoni]

Hi @RobKnop, thanks again for raising the issue. The empty qas is indeed a valid case that FARM and possibly the labelling tool(not exporting documents without answers) should deal with.