Change "Scenario" in "Benchmark" Object

Following the nomenclature described in the paper I think we should remove the notion of “Scenario” from the codebase.

At the moment, having benchmarks generators that return scenario objects is not intuitive. It is fine to use the word scenario to describe a particular setting (NC scenario, Task-Incremental scenario, etc…). However, in the classes and methods names I think this is not necessary. For example, it would be better to have NCBenchmark objects rather than NCScenario objects that offer the stream of exps.

The same goes for the generators: “nc_scenario” -> “nc_benchmark”.

What do you think?

[AntonioCarta]

I’m adding this discussion to the beta roadmap. I will ping you with a complete answer once we are ready to do the API update.

[mmasana]

The idea behind plugins was to give an easy mechanism to extend strategies by defining isolated components that can be added to any strategies. These components can be CL strategies themselves, like regularization or replay strategies, that can be more or less considered as separate components that you can add on top of a Naive strategy. But you can also have other kinds of plugins, like a plugin for pytorch learning rate schedulers, or loggers. I believe the main benefit of plugins is the fact that you can split any additional functionality inside its own class without having a large BaseStrategy with tons of arguments, with BaseStrategy providing only a basic skeleton.

That makes sense, I think the main issue is that if some methods are part plugin and part strategy (or more than one of each), it can become more difficult to understand its implementation. However, I agree that once you get past this initial phase, it can make it easier to combine or extend those parts, or repurpose them to other scenarios. The lack of understanding of how some plugins or strategies work can be detrimental since someone can be bypassing the rules/assumptions of some scenarios. But I guess with proper documentation this can also be mitigated.

Maybe restricting plugins could simplify this decision process?

My first answer would be yes, since new users could always implement new methods as strategies, and then if something becomes useful, it could be upgraded to plugin with further work on making sure it can be plugged properly in multiple situations (by the avalanche dev team). However, I do understand that would lead to more work and updates on the framework over time. In any case, having one clear way to implement them would always make things easier.

Do you have a better name? Remember to consider that plugins are not just strategies, metrics and loggers are also plugins.

It’s tricky because I would not use those names we are using right now, but I understand the nuances of coming up with smth else. Using words such as method or approach for strategies and the word strategy for plugin would be my go to. But I’m not sure it would solve the problem in some of the cases. Having metrics and loggers into plugins makes sense with the name plugins, but having them mixed with the methods and strategies is quite odd to me, and I wonder how it will look like when 40 new methods are added to the framework.

This is certainly something that will evolve over time. I’m not sure if a strict hierarchical organization is needed. For example torch.nn doesn’t really have one despite the large number of layers.

In this case, I think they have quite a good hierarchy organization with Module() as the basic building block for the network graphs, which then bifurcate into mainly layers and functions that all have the same behaviour and usage. In plugins, you can have metrics and a methods both under the same direct umbrella, which have very different usage.

This is something that we should definitely encode and clarify. However, I think some strategies may be considered “pluggable”, in the sense that you can attach them to any other strategy. For example, a pure regularization strategy can probably be safely added, like you would do with a regularization function.

I agree, but in this example, some plugins are not compatible between them, and I think this is what makes them different. If plugins are different metrics or loggers, you can use as many as you want and they will all provide their output in the corresponding shape. But multiple methods being used as plugins maybe is not so straightforward, and it would be better to fuse them under the strategy umbrella.