question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. ItΒ collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to evaluate model with current weights?

See original GitHub issue

Hi,

While I am running a pretrained CRDNN model, I finetune it on some data with .fit() method. Then, I would like to see its performance with .evaluate() function.

However, when I run the evaluate() function the log is something like this:

  0%|          | 0/16 [00:00<?, ?it/s]
  6%|β–‹         | 1/16 [00:22<05:42, 22.84s/it]
 12%|β–ˆβ–Ž        | 2/16 [00:29<03:03, 13.12s/it]
 19%|β–ˆβ–‰        | 3/16 [00:54<04:04, 18.84s/it]
 25%|β–ˆβ–ˆβ–Œ       | 4/16 [01:21<04:23, 21.98s/it]
 31%|β–ˆβ–ˆβ–ˆβ–      | 5/16 [01:47<04:18, 23.45s/it]
 38%|β–ˆβ–ˆβ–ˆβ–Š      | 6/16 [02:17<04:17, 25.71s/it]
 44%|β–ˆβ–ˆβ–ˆβ–ˆβ–     | 7/16 [02:43<03:50, 25.65s/it]
 50%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ     | 8/16 [03:11<03:31, 26.41s/it]
 56%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹    | 9/16 [03:20<02:27, 21.08s/it]
 62%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž   | 10/16 [03:26<01:37, 16.31s/it]
 69%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰   | 11/16 [03:40<01:17, 15.56s/it]
 75%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ  | 12/16 [04:08<01:18, 19.58s/it]
 81%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 13/16 [04:38<01:08, 22.67s/it]
 88%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 14/16 [04:50<00:38, 19.27s/it]
 94%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–| 15/16 [04:57<00:15, 15.83s/it]
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 16/16 [05:35<00:00, 22.38s/it]
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 16/16 [05:35<00:00, 20.98s/it]
speechbrain.utils.train_logger - Epoch loaded: 15 - test loss: 1.50, test CER: 31.43, test WER: 37.69

As you can see, it seems that model is evaluated on Epoch 15 which probably indicates that it loads weights from that epoch instead of using the weights which have just been finetuned. Whenever I do this finetuning and then evaluate(), the scores are exactly the same (up to 8 decimal digits). How do I overcome that?

TLDR; I want to use current weights with .evaluate() function, not with weights loaded from best epoch. How do I achieve that?

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7

github_iconTop GitHub Comments

1reaction
TParcolletcommented, Jul 20, 2022

(In practice, you can, I believe, just pass an argument that forces to use the latest instead of the best using the keys).

0reactions
KacperKubaracommented, Jul 20, 2022

brain.checkpointer=None works. I looked over the source code for brain.checkpointer.delete_checkpoints(num_to_keep=0) but this delets the checkpoints from the directory which is not exactly what I want to do.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Training and evaluation with the built-in methods - TensorFlow
This guide covers training, evaluation, and prediction (inference) models when using built-in APIs for training & validation (such asΒ ...
Read more >
Evaluate the Performance of Deep Learning Models in Keras
In this post, you will discover a few ways to evaluate model performance using Keras. Kick-start your project with my new book Deep...
Read more >
Step 4: Build, Train, and Evaluate Your Model
Training involves making a prediction based on the current state of the model, calculating how incorrect the prediction is, and updating theΒ ...
Read more >
keras: load saved model weights in a model for evaluation
After training, I load the model weights in to a model for evaluation, but I found the model does not give the best...
Read more >
How to use a model to do predictions with Keras - ActiveState
Specifies what layers the model contains, and how they are connected. Weights. Input parameters that influence output in a Keras model.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found