h5py 3.0 incompatibility with TensorFlow model serialization (multiple versions)

• Operating System: Linux (Ubuntu 18.04), macOS
• Python version: 3.6, 3.7
• Where Python was acquired: System (Linux), Pyenv (macOS)
• h5py version: 3.0.0
• HDF5 version: 1.12.0
• The full traceback/stack trace shown:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    model.save('test.h5')
  File "/Users/tgaddair/.venv/horovod/stable/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1171, in save
    signatures)
  File "/Users/tgaddair/.venv/horovod/stable/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/save.py", line 109, in save_model
    model, filepath, overwrite, include_optimizer)
  File "/Users/tgaddair/.venv/horovod/stable/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 103, in save_model_to_hdf5
    save_weights_to_hdf5_group(model_weights_group, model_layers)
  File "/Users/tgaddair/.venv/horovod/stable/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 625, in save_weights_to_hdf5_group
    param_dset = g.create_dataset(name, val.shape, dtype=val.dtype)
  File "/Users/tgaddair/.venv/horovod/stable/lib/python3.7/site-packages/h5py/_hl/group.py", line 143, in create_dataset
    if '/' in name:
TypeError: a bytes-like object is required, not 'str'

Here are the parameter values being passed from TensorFlow into create_dataset:

self = <Closed HDF5 group>, name = b'dense/kernel:0', shape = (3, 2)
dtype = dtype('float32'), data = None, kwds = {'track_order': False}
group = <Closed HDF5 group>

As you can see, TensorFlow is providing the name as bytes type, not a str, which seems to be the crux of the issue here.

Repro script (tested with TensorFlow 1.15.0 and 2.3.1):

import tensorflow as tf

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(2, input_shape=(3,)))
model.save('test.h5')

Rolling back to h5py==2.10.0 works fine.

[bhack]

@takluyver Do have any idea about https://github.com/tensorflow/tensorflow/pull/45380 fails?

Seems that something diverged between Linux and Win/Macosx with h5py 3.x. With exactly the same code we have now AttributeError: 'str' object has no attribute 'decode' but only on Win/Macosx

[takluyver]

I’m not aware of anything in h5py that would make it platform dependent.

You might want to use h5dump (command line tool) to see what’s really in the HDF5 file you’re trying to read.