Analyze Data

Hello, I was able to run an experiment with the HighwayEnv successfully and now I am trying to analyze the data. Unfortunately I cannot seem to figure it out. I am running the command

!python3 scripts/analyze.py run path/to/data/openaigym.episode_batch.0.953.stats.json

but I keep getting the following error

---

INFO:root:Fetching data in out/HighwayEnv/DQNAgent/run_20200413-214123_953/run/openaigym.episode_batch.0.953.stats INFO:root:Found 0 runs. Traceback (most recent call last): File “/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py”, line 2646, in get_loc return self._engine.get_loc(key) File “pandas/_libs/index.pyx”, line 111, in pandas._libs.index.IndexEngine.get_loc File “pandas/_libs/index.pyx”, line 138, in pandas._libs.index.IndexEngine.get_loc File “pandas/_libs/hashtable_class_helper.pxi”, line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item File “pandas/_libs/hashtable_class_helper.pxi”, line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: ‘episode’

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “rl-agents/scripts/analyze.py”, line 131, in <module> main() File “rl-agents/scripts/analyze.py”, line 36, in main RunAnalyzer(opts[‘<run_folder>’], out=opts[“–out”], episodes_range=episodes_range) File “rl-agents/scripts/analyze.py”, line 49, in init self.analyze() File “rl-agents/scripts/analyze.py”, line 97, in analyze self.find_best_run() File “rl-agents/scripts/analyze.py”, line 111, in find_best_run df = df[df[“episode”] == df[“episode”].max()].sort_values(criteria, ascending=ascending) File “/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py”, line 2800, in getitem indexer = self.columns.get_loc(key) File “/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py”, line 2648, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File “pandas/_libs/index.pyx”, line 111, in pandas._libs.index.IndexEngine.get_loc File “pandas/_libs/index.pyx”, line 138, in pandas._libs.index.IndexEngine.get_loc File “pandas/_libs/hashtable_class_helper.pxi”, line 1619, in pandas._libs.hashtable.PyObjectHashTable.get_item File “pandas/_libs/hashtable_class_helper.pxi”, line 1627, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: ‘episode’

---

I assume the issue with pandas stems from the fact that it says “Found 0 runs” and therefore could not pass that data on. I’m not too familiar with json files and I’ve tried running

!python3 scripts/analyze.py run path/to/data/

among many others things to get it to work but have not been successful. Could you help me figure out how to analyze the data? Are there specific files that need to be in the same folder as the stats.json?

[mkapron9]

Oh that makes sense. I was able to run the benchmarks and got some good plots. Thank you for your help. One quick question, what is the meaning of the “crashed” metric? It seems to jump around a bit between 0 and 1 with increments of 0.1. I couldn’t find how that was being calculated.

[eleurent]

Yes exactly, otherwise you get noisy plots difficult to read. Here, you can read it as a probability of having a collision in 10 episodes.