Cannot use test and reproduce the result?

@dukebw Hi, thanks for your code. I download it and run it but meet with 3 main problems:

1. It seems that it cannot reproduce the result in the paper? I run it using the run.sh by default and the eval ppl is around 80~100 until the end of the training (150 epochs)
2. There is no test function in the Trainer class. I add one using the evaluation method (by passing self.test_data as the arg). However, the ppl is around 1500. Even when I pass self.train_data, self.eval_data or self.valid_data, it`s also around 1500.
3. After training is done, when I call either test() or derive() and pass the arg --load_path, the self.shared.load_state_dict in load_model() throws out an error as “KeyError: unexpected key batch_norm.weight in state_dict”. Moreover, I print the self.shared.state_dict.keys() and the content loaded by torch.load from the checkpoint, and find that, parameters stored in the checkpoint contain 4 parameters related with batch normalization as “batch_norm.weight”, “batch_norm.bias”, “bath_norm.running_mean”, “batch_norm.running_var”, while the parameters shown by self.shared.load_state_dict not.

It would be great if you could help check these.

[dukebw]

Hi Ren Qian,

To address 1. and 2., it is necessary to re-train the model from scratch after discovering a good architecture. See Section 4.5 of this early revision of the ENAS paper: https://openreview.net/references/pdf?id=BykVS-WC-. This re-training is not implemented yet.

My results on the validation of the shared model are similar to yours, with the best validation perplexity I have seen being 82.

The third point you raised sounds like a genuine bug. I think it has to do with these lines in models/shared_rnn.py: https://github.com/carpedm20/ENAS-pytorch/blob/5736a24ef6d1967e4e9eefe2f6662a872a98586b/models/shared_rnn.py#L185-L188.

Thank you for pointing out the issues.

[philtomson]

Also, re the last reply: it’s not clear to me from reading the paper if that picking the best model of the samples and then retraining happens every epoch or at the end of all the epochs.