where is the data pipeline for training?

I do not find the info for training and testing on the kaggle dataset.

there is only one line for dataset creation in data_utils.py

getKaggleCriteoAdData(datafile="<path-to-train.txt>", o_filename=kaggleAdDisplayChallenge_processed.npz")

however, in the dlrm_s_pytorch.py, there is no where can input a dataset.

    parser.add_argument("--data-set", type=str, default="kaggle")  # or terabyte
    parser.add_argument("--raw-data-file", type=str, default="")
    parser.add_argument("--processed-data-file", type=str, default="")

these 3 lines above are useless, since no ref in the code for these parameters.

what is the purpose to publish the source code that is not able to read any dataset?

[insoulity]

FYI, I am NOT one of the authors. But let me add a few comments in the meantime.

First of all, I’d like to say I and my colleagues have been completely fine with running DLRM (while the one before refactoring data loader) based on the readme.

I agree that the script could have had --raw-data-file by default but it’s also well described in the https://github.com/facebookresearch/dlrm#benchmarking.

For the earlier one, could you take a look at the following to get to data_utils.py?

• https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L500
• https://github.com/facebookresearch/dlrm/blob/master/dlrm_data_pytorch.py#L42
• https://github.com/facebookresearch/dlrm/blob/f57a80f5e23a3ae15260342179fcc558d345dece/data_utils.py#L292

Hope this helps meanwhile. Yongkee

[mnaumovfb]

Please refer to the detailed description in part 2 of the benchmarking section in the README file.

We do not distribute any datasets with this model. We do support interface with Kaggle Display Advertising Challenge Dataset. In order to use it:

1. Please download the dataset yourself. Then, you will see a raw data file train.txt
2. Please pass this file to the model using “–raw-data-file=…” command line argument
3. The model will generate a processed .npz file, which can be used in subsequent runs
4. Please refer to the ./bench/dlrm_s_criteo_kaggle.sh script for how it is used.