Evaluate for Runner
See original GitHub issue🚀 Feature Request
The evaluate_loader
method for Python API. Similar to .train
and .predict_loader
Motivation
Proposal
Possible use case
import os
from torch import nn, optim
from torch.utils.data import DataLoader
from catalyst import dl, utils
from catalyst.data.transforms import ToTensor
from catalyst.contrib.datasets import MNIST
model = nn.Sequential(nn.Flatten(), nn.Linear(28 * 28, 10))
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.02)
loaders = {
"train": DataLoader(
MNIST(os.getcwd(), train=True, download=True, transform=ToTensor()), batch_size=32
),
"valid": DataLoader(
MNIST(os.getcwd(), train=False, download=True, transform=ToTensor()), batch_size=32
),
}
runner = dl.SupervisedRunner(
input_key="features", output_key="logits", target_key="targets", loss_key="loss"
)
# model training
runner.train(
model=model,
criterion=criterion,
optimizer=optimizer,
loaders=loaders,
num_epochs=1,
callbacks=[
dl.AccuracyCallback(input_key="logits", target_key="targets", topk_args=(1, 3, 5)),
dl.PrecisionRecallF1SupportCallback(
input_key="logits", target_key="targets", num_classes=10
),
dl.AUCCallback(input_key="logits", target_key="targets"),
# catalyst[ml] required ``pip install catalyst[ml]``
# dl.ConfusionMatrixCallback(input_key="logits", target_key="targets", num_classes=10),
],
logdir="./logs",
valid_loader="valid",
valid_metric="loss",
minimize_valid_metric=True,
verbose=True,
load_best_on_end=True,
)
loader_metrics = runner.evaluate_loader(
loader=loaders["valid"]),
callbacks=[
dl.AccuracyCallback(input_key="logits", target_key="targets", topk_args=(1, 3, 5)),
dl.PrecisionRecallF1SupportCallback(
input_key="logits", target_key="targets", num_classes=10
),
])
Alternatives
The whole method could be easily done with the .train
approach, but for a more user-friendly API – why should not we add a simplified alias?
Additional context
Checklist
- feature proposal description
- motivation
- extra proposal context / proposal alternatives review
FAQ
Please review the FAQ before submitting an issue:
- I have read the documentation and FAQ
- I have reviewed the minimal examples section
- I have checked the changelog for main framework updates
- I have read the contribution guide
- I have joined Catalyst slack (#__questions channel) for issue discussion
Issue Analytics
- State:
- Created 2 years ago
- Comments:17 (17 by maintainers)
Top Results From Across the Web
The Runner's Evaluation - Focus Physical Therapy
Accurately assess running-specific movement patterns, mobility and strength deficits,; Evaluate dynamic posture and motor control and; Assess running stability.
Read more >Basic Assessments and Movement Evaluations for Runners
The first step to any effective training or prehab/rehab program is to evaluate the athlete or client in regards to Movement Quality and...
Read more >Runner Evaluation | totalphysicaltherapy
The Running Evaluation is an assessment provided by a licensed physical therapist designed to give you insight on how you move as a...
Read more >Running Assessment: For Runners at All Levels
Running history: We gather information about footwear and running distance and perform an evaluation of lower extremity and core muscle strength ...
Read more >Running Evaluations & Gait Analysis
Running is a whole-body activity. A gait analysis provides a holistic view of how your body moves and how this movement can affect...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
btw, it would be also great if we could update our main
Readme.md
example withrunner.evaluate_loader
😃 Looking forward to see your contribution 😉It looks good from my point of view 👍 The only thing, that would be definitely required: different test cases. For example, I am not sure we need to use
CheckpointCallback
(which is usually turned on iflogdir
is specified) to save our model. Or we should write proper docs to "Do not mess train and evaluate logdirs 😃 ", something like that.