loss function (In Policy Gradient section), optimizer and entropy

Dear Mr.hongzi I was interested in your resource scheduling method. Now, I stuck in your network class. I can’t understand why you used the blow function: loss = T.log(prob_act[T.arange(N), actions]).dot(values) / N Did you calculate the special loss function? If you didn’t, what’s the name of this loss function?

[hongzimao]

You are right that rmsprop_updates is a customized function. I guess back at that time standardized library for those optimizers were not available 😃 Things are easier nowadays. And you are right about the gradient operations in tensorflow or pytorch.

[hongzimao]

Here’s how we computed the advantage Gt, with a time-based baseline: https://github.com/hongzimao/deeprm/blob/master/pg_re.py#L193-L202.

IIRC, RMSProp was slightly more stable than Adam in our experiment. FWIW, A3C original paper also used RMSProp (https://arxiv.org/pdf/1602.01783.pdf see Optimizations in section 4).

The last comment was about different episode termination criteria. It’s their literal meaning I think, ‘no new jobs’ ends the episode when no new jobs are coming and ‘all_done’ only terminates the episode when all jobs (including the unfinished ones when ‘no_new_jobs’ is satisfied) are completed: https://github.com/hongzimao/deeprm/blob/b42eff0ab843c83c2b1b8d44e65f99440fa2a543/environment.py#L255-L265.