how to discrete the state

Hi,Cihan Ceyhan. I have learned your code of flappybird-qlearning-bot, it’s excellent. But I have one questions about these codes, could you help me?

how to discrete the state? ` def map_state(self, xdif, ydif, vel): ‘’’ Map the (xdif, ydif, vel) to the respective state, with regards to the grids The state is a string, “xdif_ydif_vel”

    X -> [-40,-30...120] U [140, 210 ... 420]
    Y -> [-300, -290 ... 160] U [180, 240 ... 420]
    '''
    if xdif < 140:
        xdif = int(xdif) - (int(xdif) % 10)
    else:
        xdif = int(xdif) - (int(xdif) % 70)

    if ydif < 180:
        ydif = int(ydif) - (int(ydif) % 10)
    else:
        ydif = int(ydif) - (int(ydif) % 60)

    return str(int(xdif))+'_'+str(int(ydif))+'_'+str(vel)`

how do you determine the boundary of X and Y? Thank you very much.

[ProQianXiao]

@chncyhn Thanks for your reply, now I have a deeper understanding. I will read the 'comparison of Q-learning vs SARSA ’ carefully, thanks for your help again, that means a lot for me .

[chncyhn]

@ProQianXiao yes your understanding is mostly correct. (except the “…no matter what action it took” statement, because we punish the (state,action) pairs, not just the state; note that we update self.qvalues[state][act]).

Another small difference is that you said “… it means that the bird crashed on the upper pipe, so the states before crashing that took ‘flap’ action should be punished”, but only the last state where ‘flap’ action is taken is punished, specifically the last (state, action) pair where action is ‘flap’.

I haven’t looked much into it, but in this thread there is a good comparison of Q-learning vs SARSA which you might find useful.