Cannot reproduce the Privileged Agent reported results

I’m trying to reproduce the privileged agent by training and benchmarking it on my dataset. I generated the dataset by running Carla 0.9.6 and data_collector.py file with the default parameters. However, despite what you’ve mentioned in your paper, I had to generate 200 episodes for the training set to get 179103 frames! So my first question is how did you generate 174k frames with only 100 episodes? For the validation set, I’ve generated 20 episodes in the same town (town01) as training with 18188 frames.

Here is the train/val loss that I got so far:

As you can see, I can’t get the validation loss smaller or close to 3e-5 as you mentioned on the README page. I’ve benchmarked the agent on both 128th and 256th checkpoints in Town02 but the results are far worse than what you’ve reported.

[dotchen]

Also if you don’t mind you can open a new issue to discuss this, or take the conversation on email.

[dotchen]

Is it fair to say that the focal vehicle runs a different autopilot policy compared to other vehicle in the scene?

That’s exactly what happens. In CARLA 0.9.6 other vehicles’ controllers are implemented on the server/C++ side.

I am not sure why it would run red lights though.

How often does this happen? Last time I checked this does not happen except there is one traffic light constantly ignored because it is mislabeled on the CARLA map… Let me know if this is not the case though.