Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Change "Scenario" in "Benchmark" Object

See original GitHub issue

Following the nomenclature described in the paper I think we should remove the notion of “Scenario” from the codebase.

At the moment, having benchmarks generators that return scenario objects is not intuitive. It is fine to use the word scenario to describe a particular setting (NC scenario, Task-Incremental scenario, etc…). However, in the classes and methods names I think this is not necessary. For example, it would be better to have NCBenchmark objects rather than NCScenario objects that offer the stream of exps.

The same goes for the generators: “nc_scenario” -> “nc_benchmark”.

What do you think?

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (3 by maintainers)

github_iconTop GitHub Comments

AntonioCartacommented, May 14, 2021

I’m adding this discussion to the beta roadmap. I will ping you with a complete answer once we are ready to do the API update.

mmasanacommented, May 13, 2021

The idea behind plugins was to give an easy mechanism to extend strategies by defining isolated components that can be added to any strategies. These components can be CL strategies themselves, like regularization or replay strategies, that can be more or less considered as separate components that you can add on top of a Naive strategy. But you can also have other kinds of plugins, like a plugin for pytorch learning rate schedulers, or loggers. I believe the main benefit of plugins is the fact that you can split any additional functionality inside its own class without having a large BaseStrategy with tons of arguments, with BaseStrategy providing only a basic skeleton.

That makes sense, I think the main issue is that if some methods are part plugin and part strategy (or more than one of each), it can become more difficult to understand its implementation. However, I agree that once you get past this initial phase, it can make it easier to combine or extend those parts, or repurpose them to other scenarios. The lack of understanding of how some plugins or strategies work can be detrimental since someone can be bypassing the rules/assumptions of some scenarios. But I guess with proper documentation this can also be mitigated.

Maybe restricting plugins could simplify this decision process?

My first answer would be yes, since new users could always implement new methods as strategies, and then if something becomes useful, it could be upgraded to plugin with further work on making sure it can be plugged properly in multiple situations (by the avalanche dev team). However, I do understand that would lead to more work and updates on the framework over time. In any case, having one clear way to implement them would always make things easier.

Do you have a better name? Remember to consider that plugins are not just strategies, metrics and loggers are also plugins.

It’s tricky because I would not use those names we are using right now, but I understand the nuances of coming up with smth else. Using words such as method or approach for strategies and the word strategy for plugin would be my go to. But I’m not sure it would solve the problem in some of the cases. Having metrics and loggers into plugins makes sense with the name plugins, but having them mixed with the methods and strategies is quite odd to me, and I wonder how it will look like when 40 new methods are added to the framework.

This is certainly something that will evolve over time. I’m not sure if a strict hierarchical organization is needed. For example torch.nn doesn’t really have one despite the large number of layers.

In this case, I think they have quite a good hierarchy organization with Module() as the basic building block for the network graphs, which then bifurcate into mainly layers and functions that all have the same behaviour and usage. In plugins, you can have metrics and a methods both under the same direct umbrella, which have very different usage.

This is something that we should definitely encode and clarify. However, I think some strategies may be considered “pluggable”, in the sense that you can attach them to any other strategy. For example, a pure regularization strategy can probably be safely added, like you would do with a regularization function.

I agree, but in this example, some plugins are not compatible between them, and I think this is what makes them different. If plugins are different metrics or loggers, you can use as many as you want and they will all provide their output in the corresponding shape. But multiple methods being used as plugins maybe is not so straightforward, and it would be better to fuse them under the strategy umbrella.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Four different benchmark scenarios used in ... - ResearchGate
Download scientific diagram | Four different benchmark scenarios used in the benchmark software framework to test generalization capabilities.
Read more >
Benchmark Factory for Database 8.0 - User Guide
In the Edit Job or New Job Wizard, under Workload select the Create Objects for benchmark step for the selected benchmark test. From...
Read more >
Bencher-Scenarios-TreeObject-0.05 - MetaCPAN
Scenarios to benchmark Tree-Object & Tree-ObjectXS. ... Changes for version 0.05 - 2017-01-25 ... Bencher::Scenario::TreeObject::descendants. Benchmark ...
Read more >
State Change Object Detection on Ego4D - Papers With Code
The current state-of-the-art on Ego4D is InternVideo. See a full comparison of 1 papers with code.
Read more >
CORe50 — Continuum 0.1.0 documentation
... in “CORe50: a new Dataset and Benchmark for Continuous Object Recognition”. ... You can create automatically scenarios with continuum by setting the ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found