question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Feature Request] custom fusion method in optimize_fusion

See original GitHub issue

Issue Description

Is your feature request related to a problem? Please describe. Hi, you’ve done a great job implementing plenty of different fusion algorithms, but I think it will always be a bottleneck. What would you think about letting the user define their own training function?

Describe the solution you’d like For example, in optimize_fusion, allow method to be a callable and in this case, do not call has_hyperparams and optimization_switch.

Describe alternatives you’ve considered

  • Open a feature request every time I want to try out something new 😃
  • Fork ranx and implement new fusion methods there

My use case/ Ma et al. By the way, at the moment, my use case is to use the default-minimum trick of Ma et al.: when combining results from systems A and B, it consists in giving the minimum score of A’s results if a given document was only retrieved by system B, and vice-versa.

Maybe this is already possible in ranx via some option/method named differently? Or maybe you’d like to add it in the core ranx fusion algorithms?

Issue Analytics

  • State:closed
  • Created 2 months ago
  • Comments:14 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
AmenRacommented, Nov 30, 2022

I never used ZMUV, to be honest. I implemented it for completeness and tried it for comparison purposes but never got better results than min-max, max, or sum, which sometimes works the best.

In general, I prefer local normalization schemes because they are “unsupervised” and can be used out of the box. Without strong empirical evidence that default-minimum (w/ or w/o ZMUV) works better than min-max, max, or sum, I would not use it.

Also, without a standardized way of normalizing/fusing results is often difficult to understand what brings improvements over the state-of-the-art. Conducting in-depth ablation studies is costly, and we often lack enough space on conference papers to write about them.

1reaction
AmenRacommented, Nov 29, 2022

Thank you very much, Paul!

I am happy to see that max-norm outperforms default-minimum. To give you some context, I added/invented max norm because the minimum score is often unknown. We usually fuse only the top retrieved documents from each model, which makes min-max (in this specific context) not very sound to me. I did not do extensive experimentation but from my experience max norm outperforms min-max very often.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Feature Request - Fusion 360 Community - Autodesk Forums
I really want to design our product parametrically using Fusion 360, but currently we are unable to do this due to the complex...
Read more >
Can I send you a feature request? | Avada Website Builder
For any feature requests, recommendations or ideas we kindly ask that you submit this at https://theme-fusion.com/contact-us.
Read more >
Feature Requests - WP Fusion
Feature Requests Got an idea for a new feature or integration? We'd love to hear it! You can find our roadmap here.
Read more >
Fusion - Blackmagic Design
Fusion is the world's most advanced compositing software for visual effects artists, broadcast and motion graphic designers and 3D animators. With over 30...
Read more >
SI-LOK® Select Sacroiliac Joint Fusion System - Globus Medical
SI-LOK® SELECT Sacroiliac Joint Fusion System ... Features and Benefits ... Overlapping slot geometry is engineered to optimize fusion across the SI joint ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found