Converting `highway-v0` to continuous control

Hello! I’d like to know what is the best way to make a version of highway-v0 with continuous controls (steering and acceleration).

Looking at the source code, I reckon that the most straightforward way would be to implement a subclass of HighwayEnv and override its step method. It would then call

self.vehicle.act(
    {"acceleration": action[0] * self.ACCELERATION_RANGE,
    "steering": action[1] * self.STEERING_RANGE}
)

and then self.simulate() with no arguments, bypassing the automatic steering and acceleration presets.

However, I’m not sure how to detect lane changes, which are necessary for calculating the reward. Any help would be appreciated, as there’s quite a bit of code to look at.

Great job with the simulations btw!

[eleurent]

That looks great! I chose to use the target lane/velocity instead of the true state in the reward function so as to avoid skipping a reward because of a potential lag in the control response, but your changes make a lot of sense in the case of a Vehicle. I’m closing this issue but please, definitely feel free to open another one if you have more questions! ALSO: if you come up with a new interesting environment, please consider opening a pull request 😃 Edouard

[0xangelo]

Calling Vehicle.create_random totally works! This is what I’m doing now:

    @override(HighwayEnv)
    def _create_vehicles(self):
        self.vehicle = Vehicle.create_random(self.road, 0)
        self.road.vehicles.append(self.vehicle)
        ...

I also had to update _reward, since Vehicle does not have some attributes that MDPVehicle has.

    @override(HighwayEnv)
    def _reward(self, _):
        idx = self._curr_lane()
        action_reward = self.LANE_CHANGE_REWARD if idx != self._old_lane else 0
        self._old_lane = idx
        neighbours = self.road.network.all_side_lanes(self.vehicle.lane_index)
        state_reward = (
            +self.config["collision_reward"] * self.vehicle.crashed
            + self.RIGHT_LANE_REWARD
            * self.vehicle.lane_index[2]   # Changed from vehicle.target_lane_index
            / (len(neighbours) - 1)
            + self.HIGH_VELOCITY_REWARD
            * MDPVehicle.speed_to_index(self.vehicle.velocity)  # Changed from vehicle.velocity_index
            / (MDPVehicle.SPEED_COUNT - 1)
        )
        return utils.remap(
            action_reward + state_reward,
            [
                self.config["collision_reward"],
                self.HIGH_VELOCITY_REWARD + self.RIGHT_LANE_REWARD,
            ],
            [0, 1],
        )

Again, thanks a lot for your help!