question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Metrics general supplementary checking before `compute`

See original GitHub issue

As discussed here, the following assumption:

All metrics that has not been updated yet after last reset should raise NotComputableError when compute is called

holds for any metric. Today we check metric specific variable _num_examples if it’s equal zero and raise exception if it is the case. This behaviour can be replaced directly in Metric by counting the number of update calls between reset and before compute.
An example implementation is proposed by @zasdfgbnm :

class Metric:
    def  __init__(self, ...):
        ......

        def wrapped_reset():
            self._updated = False
            self.reset()
        self.reset = functools.wraps(self.reset)(wrapped_reset)

        def wrapped_update(output):
            self._updated = True
            self.update(output)
        self.update =  functools.wraps(self.update)(wrapped_update)

        def wrapped_compute():
            if not self._updated:
                raise NotComputableError('not updated before compute')
            return self.compute()
        self.compute =  functools.wraps(self.compute)(wrapped_compute)

Let’s discuss here what we can do.

cc @zasdfgbnm

Issue Analytics

  • State:open
  • Created 5 years ago
  • Comments:19 (11 by maintainers)

github_iconTop GitHub Comments

1reaction
zasdfgbnmcommented, Nov 18, 2018

@vfdev-5 It is a little bit similar, but for this the operands are classes like Precision, and the result are also class. The idea is, if A and B are classes for metrics, then A + B is another class for a new metric. Should be something like

class MetricMeta(type):

    def __add__(cls1, cls2):
        if "signature of __init__ of cls1 and cls2 does not match":
            raise "can not add cls1 and cls2"

        class Result(Metric):

            def __init__(self, *args, **kwargs):
                self.obj1 = cls1(*args, **kwargs)
                self.obj2 = cls2(*args, **kwargs)

            def reset(self):
                self.obj1.reset()
                self.obj2.reset()

            def update(self, output):
                self.obj1.update(output)
                self.obj2.update(output)

            def compute(self):
                return self.obj1.compute() + self.obj2.compute()

        return Result
1reaction
zasdfgbnmcommented, Nov 16, 2018

@vfdev-5 I opened a new issue for the metaclass problem: https://github.com/pytorch/ignite/issues/337

Read more comments on GitHub >

github_iconTop Results From Across the Web

Supplementary Guidelines | OpenTelemetry
For each metric stream, two measurements are produced covering the same interval of time, which the SDK is expected to aggregate before producing...
Read more >
Picard Metrics Definitions - GitHub Pages
ClusteredCrosscheckMetric: A Metric class to hold the result of clustered crosschecking fingerprints. CollectHiSeqXPfFailMetrics.PFFailDetailedMetric: a metric ...
Read more >
Recruiting Metrics Cheat Sheet - LinkedIn Business
*Some companies use a benchmark of 3-4 interviewed candidates before a candidate is hired. Recruiting Metrics Cheatsheet. 5 talent.linkedin.com. The candidates ...
Read more >
Supplementary information - Nature
General information. Supplementary Information (SI) is peer-reviewed material directly relevant to the conclusion of a paper that cannot be included in the ...
Read more >
11 Most Important Website Metrics You Should Track - HostPapa
Metrics can also indicate issues with your website, a particular product web page or general performance. Your website's metrics tell a lot ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found