train ipet with zero training examples

Hi, I am training ipet with zero training examples, I run the following command. python3 cli.py --method ipet --pattern_ids 0 1 2 3 4 --data_dir /share/home/zqzeng/wmni/data/ag_news_csv/ag_news_csv --model_type roberta --model_name_or_path /share/home/zqzeng/transformers/roberta-large --task_name agnews --output_dir /share/home/zqzeng/wmni/data/output/unsupervised-ipet --do_train --do_eval --pet_repetitions 1 --ipet_n_most_likely 100 --reduction mean --train_examples 0 And I got the following result: 2021-11-09 20:22:31,904 - INFO - tasks - Creating features from dataset file at ag_news_csv/ (num_examples=0, set_type=train) 2021-11-09 20:22:34,978 - INFO - tasks - Returning 120000 train examples with label dist.: [('3', 30000), ('4', 30000), ('2', 30000), ('1', 30000)] I followed the flow of the program and found that the whole train examples(120000) was uesd to train each individual model. When I used “–train_examples 10”, it’s normal, as shown below: 2021-11-09 20:19:13,402 - INFO - tasks - Creating features from dataset file at ag_news_csv/ (num_examples=10, set_type=train) 2021-11-09 20:19:16,127 - INFO - tasks - Returning 10 train examples with label dist.: [('1', 3), ('4', 4), ('2', 2), ('3', 1)] Does the zero training examples don’t work? I would be grateful for your prompt reply.

[EneruMin]

The results of each iteration are as shown below. g0

acc-p0: 0.6531578947368422 +- 0
acc-p1: 0.7471052631578947 +- 0
acc-p2: 0.5906578947368422 +- 0
acc-p3: 0.7082894736842106 +- 0
acc-p4: 0.7942105263157895 +- 0
acc-all-p: 0.6986842105263158 +- 0.07953688886139995

g1

acc-p0: 0.7794736842105263 +- 0
acc-p1: 0.7235526315789473 +- 0
acc-p2: 0.6317105263157895 +- 0
acc-p3: 0.7476315789473684 +- 0
acc-p4: 0.7796052631578947 +- 0
acc-all-p: 0.7323947368421052 +- 0.06101826675772052

g2

acc-p0: 0.8057894736842105 +- 0
acc-p1: 0.7156578947368422 +- 0
acc-p2: 0.7773684210526316 +- 0
acc-p3: 0.8132894736842106 +- 0
acc-p4: 0.7723684210526316 +- 0
acc-all-p: 0.7768947368421053 +- 0.03850371874690442

final

acc-p0: 0.8228947368421052 +- 0
acc-all-p: 0.8228947368421052 +- 0

According to the figure 4 in your paper, I think maybe I should use 4 or 5 iterations.

[timoschick]

Interesting… I’ll check why we didn’t get a similar error as soon as I find the time. Regardless, the final accuracy should be much better than the one you’ve reported. There are a couple of differences between your command and the one that we have used, so I cannot tell what exactly causes the difference. Could you tell me the results after each iteration (the contents of the result_test.txt file in each iteration’s directory)? That will help to identify the point where things diverge.

A couple of notes regarding possible differences:

• We have trained 3 models per pattern, so --pet_repetitions 3, whereas you’ve been using --pet_repetitions 1. Using more models stabilizes results.
• We’ve been using a batch size of 16.
• We’ve been using --lm_training (with a ratio of 1:3 training examples and unlabeled examples).
• We’ve been using 40,000 unlabeled examples. It’s not clear from your script how many unlabeled examples you’ve been using?
• I don’t remember the exact number of iterations that we’ve been using, but I think it was more than the default 3 (--ipet_generations). You can check that in our paper.

Finally, you may get different results due to random selection of examples and model initialization (but those should not account for more than 5% difference in performance). If you want to reproduce our exact results and none of the above helps, you can check out the v1.1.0 branch that contains the script that we have used for iPET.