Improve __getattr__ for VecEnvWrapper

In more complicated experiments, a user may wish to wrap a VecEnv with multiple instances of VecEnvWrapper in a modular fashion. For example, we could have something like below:

class RewardVecWrapper(VecEnvWrapper):
    def __init__(self, venv, reward_multiplier):
        super().__init__(venv)
        self._reward_multiplier = reward_multiplier

    def step_wait(self):
        obs, rewards, dones, infos = self.venv.step_wait()
        rewards *= self._reward_multiplier
        return obs, rewards, dones, infos

    def get_reward_multiplier(self):
        return self._reward_multiplier

class LoggingVecWrapper(VecEnvWrapper):
     def custom_log(self, infos):
           # stateful logging, saving to file in custom format, etc. 
           ...

     def step_wait(self):
         obs, rewards, dones, infos = self.venv.step_wait()
         self.custom_log(infos)
         return obs, rewards, dones, infos

The issue arises if I have a venv which has been wrapped with RewardVecWrapper and and I wish to call get_reward_multiplier. Depending on if I have wrapped it with LoggingVecWrapper (or any number of wrappers really), I would have to do some unappealing type-checking, something to the effect of

def get_arbitrarily_deep_reward_multiplier(wrapped_venv):
    reward_venv = wrapped_venv
    while True:
        if isinstance(reward_venv, RewardVecWrapper):
            return reward_venv.get_reward_multiplier()
        else:
            reward_venv = reward_venv.venv

I believe we can fix this by modifying VecEnvWrapper.__getattr__ to the following:

# in class VecEnvWrapper
def __getattr__(self, name):
    # Edit: line below was originally "return self.venv.__getattr__(name)"
    return getattr(self.venv, name)

This way, as long as different VecEnvWrappers have unique names for attributes of interest, one can always find that attribute from a top-level call to venv.__getattr__.

[kantneel]

__getattr__ is called when the attribute is not found for that object. This is in contrast to __getattribute__, which is called every time an attribute reference is made.

[AdamGleave]

A warning wouldn’t actually break code, but I appreciate the concern.