tf 2.4.1 bugs
See original GitHub issuewhen training using tf 2.4.1 ,
training code
CUDA_VISIBLE_DEVICES=0 python examples/multiband_melgan/train_multiband_melgan.py \ --train-dir ./dump/train/ \ --dev-dir ./dump/valid/ \ --outdir ./examples/multiband_melgan/exp/train.multiband_melgan.v1/ \ --config ./examples/multiband_melgan/conf/multiband_melgan.v1.yaml \ --use-norm 1 \
The following error occurred
2021-04-20 13:30:56.974771: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:656] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_3" op: "FlatMapDataset" input: "TensorDataset/_1" input: "Const/_2" attr { key: "Targuments" value { list { type: DT_STRING } } } attr { key: "f" value { func { name: "__inference_Dataset_flat_map_flat_map_fn_31" } } } attr { key: "output_shapes" value { list { shape { unknown_rank: true } shape { unknown_rank: true } shape { unknown_rank: true } } } } attr { key: "output_types" value { list { type: DT_STRING type: DT_STRING type: DT_STRING } } } . Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new
tf.data.Options()object then setting
options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATAbefore applying the options object to the dataset via
dataset.with_options(options). 2021-04-20 13:30:56.997978: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:656] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_3" op: "FlatMapDataset" input: "TensorDataset/_1" input: "Const/_2" attr { key: "Targuments" value { list { type: DT_STRING } } } attr { key: "f" value { func { name: "__inference_Dataset_flat_map_flat_map_fn_538" } } } attr { key: "output_shapes" value { list { shape { unknown_rank: true } shape { unknown_rank: true } shape { unknown_rank: true } } } } attr { key: "output_types" value { list { type: DT_STRING type: DT_STRING type: DT_STRING } } } . Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new
tf.data.Options()object then setting
options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATAbefore applying the options object to the dataset via
dataset.with_options(options).
Issue Analytics
- State:
- Created 2 years ago
- Comments:8 (4 by maintainers)
@k-grygiel thanks! the bugs fixed !
Try adding this in audio_mel_dataset.py