Anyone able to load tfrecords into TFRS generated with the spark-to-tf-records connector?

Anyone see this kind of error when trying to load TF records generated from Spark by the spark to tf records connector or linkedin’s spark tf record library?

Error: Error when deserializing tfrecord’s in TF 2.x: Only integers, slices (:), ellipsis (...), tf.newaxis (None) and scalar tf.int32/tf.int64 tensors are valid indices

Filed tickets there with details

• https://github.com/linkedin/spark-tfrecord/issues/19
• https://github.com/tensorflow/ecosystem/issues/178

Really just doing a simple thing, using the small movielens dataset:

    # Code for the connector
    movies_df.write.format("tfrecords").mode("overwrite").save(tf_movies_dir)
    ratings_df.write.format("tfrecords").mode("overwrite").save(tf_ratings_dir)

    # Alternatively, code for the spark to tfrecord
    movies_df.write.format("tfrecord").mode("overwrite").option("recordType", "Example").save(tf_movies_dir)
    ratings_df.write.format("tfrecord").mode("overwrite").option("recordType", "Example").save(tf_ratings_dir)

    s3 = boto3.resource("s3", verify=False)
    bucket = s3.Bucket("mybucket")

    filenames = []
    for object_summary in bucket.objects.filter(
            Prefix=f"emr/spark_apps/myapp/movielens-100k-conversion/movies-0001/part"
    ):
        filenames.append(os.path.join("s3://audiomack-master-airflow/", object_summary.key))
    movies_dataset = tf.data.TFRecordDataset(filenames)

    filenames = []
    for object_summary in bucket.objects.filter(
            Prefix=f"emr/spark_apps/myapp/movielens-100k-conversion/ratings-0001/part"
    ):
        filenames.append(os.path.join("s3://audiomack-master-airflow/", object_summary.key))
    ratings_dataset = tf.data.TFRecordDataset(filenames)

[maciejkula]

Have you looked at the docs for reading TFRecord files containing tf.train.Examples?

It looks like you’re skipping the deserialization step (converting the serialized tf.train.Example protos to dictionaries of tensors).

[Data-Jack]

@dgoldenberg-audiomack Yeah, I will do. My first guess was it was how it was being written.