Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[bug] Performance of parameterized search worse than regular search

See original GitHub issue

After upgrading to version 1.30.0+ of conan, the time to check for package updates has increased significantly.

Related issue: #5432 Breaking changes PR: #7673

Environment Details (include every applicable attribute)

Operating System+version: Ubuntu 20.04
Compiler+version: -
Conan version: >=1.30.0
Python version: 3.8.5

Artifactory machine

Operating System+version: Windows 10
Artifactory version: 7.12.6
Count of artifacts: ~400k
We are not using the revisions feature

Logs (Executed commands with output) (Include/Attach if Applicable)

$ curl http://URL_REDACTED:8081/artifactory/api/system/version
{
  "version" : "7.12.6",
  "revision" : "71206900",
  "addons" : [ "conan" ],
  "license" : "Artifactory Community Edition for C/C++"
}

$ time conan search "PACKAGE" -r remote
Existing package recipes:

...

real	0m0,485s
user	0m0,236s
sys	0m0,055s

$ time conan search "PACKAGE/*@tc/stable" -r remote
Existing package recipes:

...

real	0m5,918s
user	0m0,264s
sys	0m0,027s

$ time curl "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE"
{
  "results" : ...
}
real	0m0,191s
user	0m0,006s
sys	0m0,006s

$ time curl "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable"
{
  "results" : ...
}
real	0m5,889s
user	0m0,015s
sys	0m0,010s

$ export CONAN_TRACE_FILE=/tmp/conan_trace.log
$ conan install . --update

/tmp/conan_trace.log:
...
{"_action": "REST_API_CALL", "duration": 5.552937269210815, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383331.6691387, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.489734649658203, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383337.1632001, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.64190936088562, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383342.806089, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 6.204895496368408, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383349.0156999, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.324174880981445, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383354.344076, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.200056314468384, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383359.5479913, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.729648590087891, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383365.2818859, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.238299608230591, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383370.524942, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.383552074432373, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383375.9145668, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 6.12669563293457, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383382.0461524, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.694319725036621, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383387.7447596, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 4.910779237747192, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383392.6595242, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.264143705368042, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383397.9279287, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 4.740357875823975, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383402.67033, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 4.9585487842559814, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383407.6303356, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.588382005691528, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383413.2226527, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.976093292236328, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383419.2005298, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.568065166473389, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383424.7730927, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.500042200088501, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383430.2782497, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40tc%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 6.4756903648376465, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383436.7583406, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40for_testing%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 6.1877007484436035, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383442.950587, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40for_testing%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 6.225765943527222, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383449.1806679, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40for_testing%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.841764450073242, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383455.027342, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40for_testing%2Fstable&ignorecase=False"}
{"_action": "REST_API_CALL", "duration": 5.6164186000823975, "headers": {"User-Agent": "Conan/1.33.1 (Python 3.8.5) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": "USER_REDACTED"}, "method": "GET", "time": 1613383460.6494935, "url": "http://URL_REDACTED:8081/artifactory/api/conan/conan-local/v1/conans/search?q=PACKAGE%2F%2A%40for_testing%2Fstable&ignorecase=False"}
...

Issue Analytics

State:
Created 3 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

c0rtcommented, Feb 16, 2021

Thanks @memsharded for quick response.

Yes, we are using the embedded Derby DB. I will try to migrate to Postresql later and give you feedback.

0reactions

memshardedcommented, May 14, 2021

Hi @c0rt

Very happy to hear that the issue improved migrating to PostgreSQL! We will report on the JIRA ticket as well.

Thanks for following up and reporting 😃

Top Results From Across the Web

Why my Parameter Tuning gives worse results? - Kaggle

You can try either Hyperopt or Halving randomized search technique, both are very optimized and 10 times faster than normal Brute force (...

Why does a parameterized query produces vastly slower ...

In a SQL Server 2005 database I'm working on this query: select * from foo

Is it possible to get worse model after optimization?

Yes, theoretically, by pure luck, it is possible that your initial guess, before optimization of hyper-parameters, provides better results than ...

Neural Parameter Allocation Search - OpenReview

We introduce Neural Parameter Allocation Search (NPAS), a novel task where ... 9, "Emb" shows worse performance than "Random" and the advantage of...

Ray Tune FAQ — Ray 2.2.0 - the Ray documentation

Search algorithm: Search algorithms have to be seeded to generate the same hyperparameter configurations in each run. Some search algorithms can be explicitly ......