Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
TypeError: '>=' not supported between instances of 'int' and 'str' in automl job
See original GitHub issueDescribe the bug
I am trying to run an automl job against a remote Ray cluster. The interesting thing is the same job can run locally but when it connects to remote Ray cluster. It’s failed because of the TypeError.
script.py
import logging
import pprint
from ludwig.datasets import mushroom_edibility
from ludwig.automl import auto_train
mushroom_edibility_df = mushroom_edibility.load()
auto_train_results = auto_train(
dataset=mushroom_edibility_df,
target='class',
time_limit_s=3600,
tune_for_memory=False,
user_config={'hyperopt': {'executor': {'num_samples': '10'},'goal': 'maximize', 'metric': 'accuracy', 'output_feature': 'class'}},
)
pprint.pprint(auto_train_results)
root@260b50ef28e7:/tmp/dataset# export RAY_ADDRESS=ray://10.227.151.166:32471
root@260b50ef28e7:/tmp/dataset# python3 script.py
NumExpr defaulting to 4 threads.
Using address ray://10.227.151.166:32471 set in the environment variable RAY_ADDRESS
Passing the following kwargs to ray.init() on the server: ignore_reinit_error
{ 'executor': { 'cpu_resources_per_trial': 1,
'num_samples': '10',
'scheduler': { 'grace_period': 72,
'max_t': 3600,
'reduction_factor': 5,
'time_attr': 'time_total_s',
'type': 'async_hyperband'},
'time_budget_s': 3600,
'type': 'ray'},
'goal': 'maximize',
'metric': 'accuracy',
'output_feature': 'class',
'parameters': { 'combiner.bn_momentum': { 'categories': [ 0.4,
0.3,
0.2,
0.1,
0.05,
0.02],
'space': 'choice'},
'combiner.bn_virtual_bs': { 'categories': [ 256,
512,
1024,
2048,
4096],
'space': 'choice'},
'combiner.num_steps': { 'categories': [ 3,
4,
5,
6,
7,
8,
9,
10],
'space': 'choice'},
'combiner.output_size': { 'categories': [ 8,
16,
24,
32,
64,
128],
'space': 'choice'},
'combiner.relaxation_factor': { 'categories': [ 1.0,
1.2,
1.5,
2.0],
'space': 'choice'},
'combiner.size': { 'categories': [8, 16, 24, 32, 64],
'space': 'choice'},
'combiner.sparsity': { 'categories': [ 0.0,
1e-06,
0.0001,
0.001,
0.01,
0.1],
'space': 'choice'},
'trainer.batch_size': { 'categories': [ 256,
512,
1024,
2048,
4096,
8192],
'space': 'choice'},
'trainer.decay_rate': { 'categories': [0.8, 0.9, 0.95],
'space': 'choice'},
'trainer.decay_steps': { 'categories': [ 500,
2000,
8000,
10000,
20000],
'space': 'choice'},
'trainer.learning_rate': { 'categories': [ 0.005,
0.01,
0.02,
0.025],
'space': 'choice'}},
'search_alg': {'random_state_seed': 42, 'type': 'hyperopt'},
'split': 'validation'}
(run pid=227) 2022-06-15 16:23:33,839 INFO trial_runner.py:508 -- No local checkpoint was found. Ray Tune will now start a new experiment.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/ludwig/hyperopt/execution.py", line 704, in execute
analysis = tune.run(
File "/usr/local/lib/python3.8/site-packages/ray/tune/tune.py", line 396, in run
return ray.get(remote_future)
File "/usr/local/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return getattr(ray, func.__name__)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/ray/util/client/api.py", line 43, in get
return self.worker.get(vals, timeout=timeout)
File "/usr/local/lib/python3.8/site-packages/ray/util/client/worker.py", line 433, in get
res = self._get(to_get, op_timeout)
File "/usr/local/lib/python3.8/site-packages/ray/util/client/worker.py", line 461, in _get
raise err
types.RayTaskError(TypeError): ray::run() (pid=227, ip=172.20.222.152)
File "/usr/local/lib/python3.8/site-packages/ray/tune/tune.py", line 671, in run
while not runner.is_finished() and not state[signal.SIGINT]:
File "/usr/local/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 687, in is_finished
return trials_done and self._search_alg.is_finished()
File "/usr/local/lib/python3.8/site-packages/ray/tune/suggest/search_generator.py", line 143, in is_finished
return self._counter >= self._total_samples or self._finished
TypeError: '>=' not supported between instances of 'int' and 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "script.py", line 11, in <module>
auto_train_results = auto_train(
File "/usr/local/lib/python3.8/site-packages/ludwig/automl/automl.py", line 119, in auto_train
return train_with_config(dataset, config, output_directory=output_directory, random_seed=random_seed, **kwargs)
File "/usr/local/lib/python3.8/site-packages/ludwig/automl/automl.py", line 205, in train_with_config
hyperopt_results = _train(
File "/usr/local/lib/python3.8/site-packages/ludwig/automl/automl.py", line 304, in _train
hyperopt_results = hyperopt(
File "/usr/local/lib/python3.8/site-packages/ludwig/hyperopt/run.py", line 307, in hyperopt
hyperopt_results = hyperopt_executor.execute(
File "/usr/local/lib/python3.8/site-packages/ludwig/hyperopt/execution.py", line 732, in execute
raise RuntimeError(f"Encountered Ray Tune error: {e}")
RuntimeError: Encountered Ray Tune error: ray::run() (pid=227, ip=172.20.222.152)
File "/usr/local/lib/python3.8/site-packages/ray/tune/tune.py", line 671, in run
while not runner.is_finished() and not state[signal.SIGINT]:
File "/usr/local/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 687, in is_finished
return trials_done and self._search_alg.is_finished()
File "/usr/local/lib/python3.8/site-packages/ray/tune/suggest/search_generator.py", line 143, in is_finished
return self._counter >= self._total_samples or self._finished
TypeError: '>=' not supported between instances of 'int' and 'str'
To Reproduce Steps to reproduce the behavior:
- Build a new container image based on
ludwigai/ludwig:0.5.2and override the entrypoint (I tried to install ludwig python package on top of ray image but encounter issues when building horovod) - Use the image to create a Ray cluster through KubeRay and get connection string
10.227.151.166:32471 - Start a new ludwig container (can access above address) and run the program.
export RAY_ADDRESS=ray://10.227.151.166:32471
python3 script.py
Expected behavior We should not see the type error.
Screenshots If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
- Python version 3.8
- Ludwig version 0.5.2
Additional context Add any other context about the problem here.
Issue Analytics
- State:
- Created a year ago
- Comments:9
Top Results From Across the Web
'>' not supported between instances of 'str' and 'int'
The “typeerror: '>' not supported between instances of 'str' and 'int'” error is raised when you try to compare a string to an...
Read more >TypeError: '<=' not supported between instances of 'str' and 'int'
I'm learning python and working on exercises. One of them is to code a voting system to select the best player between 23...
Read more >TypeError: '>' not supported between instances of 'NoneType ...
I change positives_momentum to -1, but still have error. [1,0]:Traceback (most recent call last): [1,0]: File "/root/efficientdet/pyenv/ ...
Read more >' not supported between instances of 'int' and 'str'' (Python ...
How do you resolve the error "typeerror: '<=' not supported between instances of 'int' and 'str'" (Python, Python 3.x, development)?. All related (33)....
Read more >Random Forest vs AutoML (with python code) - MLJAR
To fix, this problem, I just add new integer for new categorical values. Such new categoricals check should be implemented in the production ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hi @msuzen, can you confirm that #2286 addresses the issue?
@connor-mccorm, we may need to release a hotfix v0.5.5 if this fix isn’t included.
@connor-mccorm @ksbrar definitely something we should add to the config schema to prevent setting this value to a string.