[CLI]: Repord Editing API does not create a valid report

Describe the bug

Hello, I used python sdk API to create a report in https://wandb.ai/costa-huang/cleanRL/reports/Atari-CleanRL-PPO-gym-vs-envpool--VmlldzoyODg0NzU2, but it cannot display the lines between two groups properly. The issue is rather weird — the following video demonstrates it

https://user-images.githubusercontent.com/5555347/199073850-a805b4a3-7219-497d-bc9b-ed584535f20f.mov

A couple of notes / bugs:

1. Only the group of runs in CleanRL's ddpg_continuous_action_jax (v1.0.0b2-9-g4605546) displays correctly but not the group of runs in CleanRL's ddpg_continuous_action_jax (latest).
2. The group of runs in CleanRL's ddpg_continuous_action_jax (v1.0.0b2-9-g4605546) is being displayed twice (both the dotted and solid lines)
3. When trying to edit the filter through the App UI, the site crashed with the error Error 1054: Unknown column 'tags' in 'field list'
4. By manually cloning the two run sets using the App UI the runs are somehow displayed correctly… Not sure why.

CC @andrewtruong (thanks for the programmatic API for report editing — it’s of great help).

from typing import List
import wandb
import wandb.apis.reports as wb  # noqa
wandb.require("report-editing")
api = wandb.Api()

class Runset:
    def __init__(self, name: str, filters: dict, entity: str, project: str, groupby: str = ""):
        self.name = name
        self.filters = filters
        self.entity = entity
        self.project = project
        self.groupby = groupby

    @property
    def runs(self):
        return wandb.Api().runs(path=f"{self.entity}/{self.project}", filters=self.filters)

    @property
    def report_runset(self):
        return wb.RunSet(
            name=self.name,
            entity=self.entity,
            project=self.project,
            filters={"$or": [self.filters]},
            groupby=[self.groupby] if len(self.groupby) > 0 else None,
        )


def compare(
    runsets1: List[Runset],
    runsets2: List[Runset],
    env_ids: List[str],
):
    blocks = []
    for runset1, runset2, env_id in zip(runsets1, runsets2, env_ids):
        blocks += [
            wb.PanelGrid(
                runsets=[
                    runset1.report_runset,
                    runset2.report_runset,
                ],
                panels=[
                    wb.LinePlot(
                        x="global_step",
                        y=["charts/episodic_return"],
                        title=env_id,
                        title_x="Steps",
                        title_y="Episodic Return",
                        max_runs_to_show=100,
                        smoothing_factor=0.8,
                        groupby_rangefunc="stderr",
                        legend_template="${runsetName}",
                    ),
                    wb.LinePlot(
                        x="_runtime",
                        y=["charts/episodic_return"],
                        title=env_id,
                        title_y="Episodic Return",
                        max_runs_to_show=100,
                        smoothing_factor=0.8,
                        groupby_rangefunc="stderr",
                        legend_template="${runsetName}",
                    ),
                    # wb.MediaBrowser(
                    #     num_columns=2,
                    #     media_keys="videos",
                    # ),
                ],
            ),
        ]
    return blocks


if __name__ == "__main__":
    env_ids = ["Hopper-v2", "Walker2d-v2", "HalfCheetah-v2"]

    blocks = []
    runsets1 = [Runset(
        name="CleanRL's ddpg_continuous_action_jax (v1.0.0b2-9-g4605546)",
        filters={"$and": [
            {"config.env_id.value": env_id},
            {"tags": "v1.0.0b2-9-g4605546"},
            {"config.exp_name.value": "ddpg_continuous_action_jax"}
        ]},
        entity="openrlbenchmark",
        project="cleanrl",
        groupby="exp_name",
    ) for env_id in env_ids]
    runsets2 = [Runset(
        name="CleanRL's ddpg_continuous_action_jax (latest)",
        filters={"$and": [
            {"config.env_id.value": env_id},
            {"tags": "latest"},
            {"config.exp_name.value": "ddpg_continuous_action_jax"}
        ]},
        entity="openrlbenchmark",
        project="cleanrl",
        groupby="exp_name",
    ) for env_id in env_ids]

    blocks = compare(runsets1, runsets2, env_ids)
    print("saving report")
    report = wb.Report(
        project="cleanrl",
        title="Regression report",
        blocks=blocks,
    )
    report.save()
    print(f"view the generated report at {report.url}")

(cleanrl-py3.7) ➜  cleanrl_utils git:(rlops) ✗ python minmal_bug.py
wandb: WARNING This is an experimental feature -- use with caution!
saving report
view the generated report at https://wandb.ai//costa-huang/cleanRL/reports/Atari%3A-CleanRL-PPO-gym-vs-envpool--VmlldzoyODg0NzU2

Additional Files

No response

Environment

WandB version: 0.13.4

OS: Ubuntu

Python version: 3.7.1

Versions of relevant libraries:

Additional Context

No response

[MBakirWB]

Thank you @vwxyzjn for providing a link to the report and video demonstrating the issue you are running into. We will review and provide feedback by EOW.

[MBakirWB]

Hi @vwxyzjn, quick update, the new version of the reports API is expected to be released with our next client version and includes additional fixes/features to address some of the issues you ran into. Please let us know if you run into additional issues after trying it out.