Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Image classification: ValueError: could not broadcast input array from shape (227,300,3) into shape (750,1000,3)
See original GitHub issueHi, this is my model definition:
input_features:
-
name: image_path
type: image
encoder: stacked_cnn
output_features:
-
name: class
type: category
This is an excerpt from the data csv:
class,image_path
coyote,coyote/0411-city-coyote.jpg
coyote,coyote/125545-004-bb8f48b2.jpg
coyote,coyote/1a4053ab-279c-45ef-bd29-0dae546ece73-large16x9_coyote_file_photo.jpg
coyote,coyote/220px-howl_%28cropped%29.jpg
coyote,coyote/2yowhhahmrghtksn6eibtyrcw4.jpg
coyote,coyote/300px-2009-coyote-yosemite.jpg
coyote,coyote/359744-coyote-dmid1-5ef0hkv48-953x532.jpg
coyote,coyote/5a3c4beb76446.image.jpg
coyote,coyote/5aaab3b7c7218.image.jpg
This is the command:
ludwig experiment --data_csv coyo_road.csv --model_definition_file model_definition.yaml
This is the output:
ludwig v0.1.0 - Experiment
Experiment name: experiment
Model name: run
Output path: results/experiment_run_6
ludwig_version: '0.1.0'
command: ('/usr/local/bin/ludwig experiment --data_csv coyo_road.csv '
'--model_definition_file model_definition.yaml')
dataset_type: 'coyo_road.csv'
model_definition: { 'combiner': {'type': 'concat'},
'input_features': [ { 'encoder': 'stacked_cnn',
'in_memory': True,
'name': 'image_path',
'should_resize': False,
'tied_weights': None,
'type': 'image'}],
'output_features': [ { 'dependencies': [],
'loss': { 'class_distance_temperature': 0,
'class_weights': 1,
'confidence_penalty': 0,
'distortion': 1,
'labels_smoothing': 0,
'negative_samples': 0,
'robust_lambda': 0,
'sampler': None,
'type': 'softmax_cross_entropy',
'unique': False,
'weight': 1},
'name': 'class',
'reduce_dependencies': 'sum',
'reduce_input': 'sum',
'top_k': 3,
'type': 'category'}],
'preprocessing': { 'bag': { 'fill_value': '',
'format': 'space',
'lowercase': 10000,
'missing_value_strategy': 'fill_with_const',
'most_common': False},
'binary': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const'},
'category': { 'fill_value': '<UNK>',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000},
'force_split': False,
'image': {'missing_value_strategy': 'backfill'},
'numerical': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const'},
'sequence': { 'fill_value': '',
'format': 'space',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 20000,
'padding': 'right',
'padding_symbol': '<PAD>',
'sequence_length_limit': 256,
'unknown_symbol': '<UNK>'},
'set': { 'fill_value': '',
'format': 'space',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000},
'split_probabilities': (0.7, 0.1, 0.2),
'stratify': None,
'text': { 'char_format': 'characters',
'char_most_common': 70,
'char_sequence_length_limit': 1024,
'fill_value': '',
'lowercase': True,
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_symbol': '<PAD>',
'unknown_symbol': '<UNK>',
'word_format': 'space_punct',
'word_most_common': 20000,
'word_sequence_length_limit': 256},
'timeseries': { 'fill_value': '',
'format': 'space',
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_value': 0,
'timeseries_length_limit': 256}},
'training': { 'batch_size': 128,
'bucketing_field': None,
'decay': False,
'decay_rate': 0.96,
'decay_steps': 10000,
'dropout_rate': 0.0,
'early_stop': 3,
'epochs': 200,
'gradient_clipping': None,
'increase_batch_size_on_plateau': 0,
'increase_batch_size_on_plateau_max': 512,
'increase_batch_size_on_plateau_patience': 5,
'increase_batch_size_on_plateau_rate': 2,
'learning_rate': 0.001,
'learning_rate_warmup_epochs': 5,
'optimizer': { 'beta1': 0.9,
'beta2': 0.999,
'epsilon': 1e-08,
'type': 'adam'},
'reduce_learning_rate_on_plateau': 0,
'reduce_learning_rate_on_plateau_patience': 5,
'reduce_learning_rate_on_plateau_rate': 0.5,
'regularization_lambda': 0,
'regularizer': 'l2',
'staircase': False,
'validation_field': 'combined',
'validation_measure': 'loss'}}
Using full raw csv, no hdf5 and json file with the same name have been found
Building dataset (it may take a while)
Traceback (most recent call last):
File "/usr/local/bin/ludwig", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/ludwig/cli.py", line 86, in main
CLI()
File "/usr/local/lib/python3.6/site-packages/ludwig/cli.py", line 64, in __init__
getattr(self, args.command)()
File "/usr/local/lib/python3.6/site-packages/ludwig/cli.py", line 67, in experiment
experiment.cli(sys.argv[2:])
File "/usr/local/lib/python3.6/site-packages/ludwig/experiment.py", line 548, in cli
experiment(**vars(args))
File "/usr/local/lib/python3.6/site-packages/ludwig/experiment.py", line 234, in experiment
random_seed=random_seed
File "/usr/local/lib/python3.6/site-packages/ludwig/data/preprocessing.py", line 457, in preprocess_for_training
random_seed=random_seed
File "/usr/local/lib/python3.6/site-packages/ludwig/data/preprocessing.py", line 62, in build_dataset
**kwargs
File "/usr/local/lib/python3.6/site-packages/ludwig/data/preprocessing.py", line 90, in build_dataset_df
global_preprocessing_parameters
File "/usr/local/lib/python3.6/site-packages/ludwig/data/preprocessing.py", line 165, in build_data
preprocessing_parameters
File "/usr/local/lib/python3.6/site-packages/ludwig/features/image_feature.py", line 101, in add_feature_data
data[feature['name']][i, :, :, :] = img
ValueError: could not broadcast input array from shape (227,300,3) into shape (750,1000,3)
Run on a Macbook running macOS Sierra (10.12.6) using python 3.6
Issue Analytics
- State:
- Created 5 years ago
- Comments:5
Top Results From Across the Web
ValueError: could not broadcast input array from shape ...
I'm having the same issue but in my case I'm directly converting a raw-image to np.array using img_to_array(), tensorflow. Hence I don't have ......
Read more >ValueError: could not broadcast input array from shape ...
Image classification: ValueError: could not broadcast input array from shape (227,300,3) into shape (750,1000,3) #128.
Read more >ValueError: could not broadcast input array from shape (896 ...
Dear Cryosparc team, I ran into error when start 2D with one of my dataset with no obvious reason. It's K3 super-res movies...
Read more >ValueError could not broadcast input array from shape 360 ...
In the above error it shows could not broadcast input array from shape (360,270,3) into shape (360,280,3), means you are tying to exchange ......
Read more >Could not broadcast input array from shape (80,2) into shape ...
Whenever I hit the Start Training button this error comes up. I am not really sure what is wrong. Is it something wrong...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Actually, there was no real problem: I put the preprocessing info in the wrong place. With the model definition like this it works:
I was able to train a crappy image classifier (overall_accuracy 0.4375) over night, but my personal proof of concept worked. Thanks for the great work!
@meinnamehier you were actually putting the parameters in the right place, there was a little bug that made it so that it wouldn’t work and that you had to specify them like that. in the next pushes (and obviously the next v0.1.1) your previous code would be the correct way to do things, specifying those parameters within preprocessing. So don’t be surprised if your current yaml doesn’t work in the next release 😃