create_pretraining_data.py kept killed..

Hello. I’m working on pretraining BERT project using GCP (Google Cloud Platform).

Before I get started to use TPU for executing run_pretraining.py, I got stuck in creating pretraining data.

Here is the .sh scripts for create_pretraining_data.py

python3 create_pretraining_data.py \ --input_file $DATA_DIR/data_1.txt \ --output_file $OUTPUT_DIR \ --do_lower_case=True \ --do_whole_word_mask=True \ --max_seq_length 512 \ --max_predictions_per_seq 70 \ --masked_lm_prob 0.15 \ --vocab_file $VOCAB_DIR \ --codes_file $CODES_DIR \ --dupe_factor 1

The input text’s size is about 40GB but it seems too large so I splited the data into 18 files and each file’s size is about 1.2GB.

At first I tried to set dupe_factor as 10 but it also seems to raise memory issues so I just set dupe_factor as 1 and try to repeat 10 times assigning different random_seed.

Although I tried to execute create_pretraining_data.py with minimum environment, it kept killed and I’ve done only 1 file among 18 files.

It happened on both GCP and my local.

Have any idea to solve this “catastrophic” situation? This project has been delayed because of this issue and I don’t know what to do anymore…

[anshoomehra]

If this helps anyone:

I started with data file size over 3G with over 7 million sentences, VM was running out of RAM after a couple hours of run ( I had about 102G of RAM on VM), eventually leading system to the resource starvation state with weird errors.

If you do not have infinite RAM, as remediation, you can instead shard the data files like below:

split -d -l 250000 data_file.txt data_file_shard

I chose 250k lines per file, and it worked. You can try different size based on your system configuration.

Post this, I am able to generate n number of tf_trecord files. The run_pretraining.py step can take input as globs like tf_examples.tf_record* and hence this small addition step solved the issue completing over 3G of data processing in about 2-3 hours. I can share scripts if anyone still has issues on how to split and automated way to loop over n number of files creating tfrecords …

Good luck!!

[calusbr]

If this helps anyone:

I started with data file size over 3G with over 7 million sentences, VM was running out of RAM after a couple hours of run ( I had about 102G of RAM on VM), eventually leading system to the resource starvation state with weird errors.

If you do not have infinite RAM, as remediation, you can instead shard the data files like below:

split -d -l 250000 data_file.txt data_file_shard

I chose 250k lines per file, and it worked. You can try different size based on your system configuration.

Post this, I am able to generate n number of tf_trecord files. The run_pretraining.py step can take input as globs like tf_examples.tf_record* and hence this small addition step solved the issue completing over 3G of data processing in about 2-3 hours. I can share scripts if anyone still has issues on how to split and automated way to loop over n number of files creating tfrecords …

Good luck!!

Script? Thanks