JOSS Review: Benchmarks

Points related to benchmarks from: https://github.com/openjournals/joss-reviews/issues/2408#issuecomment-651542971

• Running the benchmarks requires pandas and matplotlib, but pandas and matplotlib are not listed as dependencies (or at least not on the main page)
• Running pytest benchmarks/ failed many of the tests on my Windows computer? The examples worked and the tests passed.

Points related to benchmarks from: https://github.com/openjournals/joss-reviews/issues/2408#issuecomment-655272567

Structure

In examples/ folder, you typically print the best value and best route. Please also ASSERT them. I can run them, but I don’t know if the solution is correct or not.

BTW, why is the separation between tests and examples. Why not add these examples to test and assert them.

Import Errors in tests/benchmarks/

Out of the box, all tests in benchmarks/ fail. All errors, except the one in test_ortools, are the same:

• with open(path + instance_name) as fp: E FileNotFoundError: [Errno 2] No such file or directory: ‘…/examples/benchmarks/data/P-n16-k8.vrp’ …\examples\benchmarks\cvrp_augerat.py:53: FileNotFoundError
• test_ortools error is

    from ortools.data import (
    E ModuleNotFoundError: No module named 'ortools'

• In test_toy.py, remove “from pytest import raises” --not used. same for test_greedy
• In cvrp_augeratpy, remove “from vrpy.main import VehicleRoutingProblem” --not used
• In benchmarks folder, some files have a main() function, some files don’t. Why?

Cordeau

• Running mdvrp_cordeau.py fails with raise KeyError(“Node %s missing from initial solution.” % v) KeyError: 'Node 52 missing from initial

Who? @torressa @Halvaros

[torressa]

Working on the restructuring and path fixing thing… Big commit dropping soon

[torressa]

That split makes more sense! Moving the test files there also makes sense.

benchmarks/ |_data/ |_run/ <- scripts for reproducing results (writing table and running different instances) |_tests/ <- Put tests/benchmarks/ here |_cvrp_augerat.py |_cvrptw_solomon.py