DataLossError: corrupted record at 0 [Op:IteratorGetNext]

System information

• Have I specified the code to reproduce the issue (Yes, No): Yes
• Environment in which the code is executed (e.g., Local(Linux/macOS/Windows), Interactive Notebook, Google Cloud, etc): Interactive Notebook (Google colab)
• TensorFlow version: 2.5.0
• TFX Version: 1.0.0
• Python version: 3.7.11

Describe the current behavior I’m following along with the TFX workshop lab 1 but with the following changes:

Installing the latest version of TFX in the installation cell: !pip install -q -U tfx

Commenting the following lines in the import cell:

from tfx.utils.dsl_utils import external_input
import tensorflow_transform as tft
from tensorflow_transform import coders as tft_coders
from tensorflow_transform.tf_metadata import dataset_schema
from tensorflow_transform.tf_metadata import schema_utils

Changing the solution of the ExampleGen Component part to the following as there’s no external_input:

example_gen = CsvExampleGen(_data_root)
context.run(example_gen)

Bypassing the following cells and in the statistics section, for the following cell:

train_uri = os.path.join(statistics_gen.outputs['statistics'].get()[0].uri, 
                         'Split-train')
tfrecord_filenames = [os.path.join(train_uri, name)
                      for name in os.listdir(train_uri)]
dataset = tf.data.TFRecordDataset(tfrecord_filenames)
for tfrecord in dataset.take(1):
  serialized_example = tfrecord.numpy()
  stats = statistics_pb2.DatasetFeatureStatisticsList()
  stats.ParseFromString(serialized_example)

I get the following error:

DataLossError: corrupted record at 0 [Op:IteratorGetNext]

This also happens for the output of the StatisticsGen Component. I haven’t checked other components yet.

Describe the expected behavior I guess there’s a problem with how the output is being written into files in the new version.

Other info / logs

Complete error log:

---------------------------------------------------------------------------
DataLossError                             Traceback (most recent call last)
<ipython-input-9-554bf159c60b> in <module>
      4                       for name in os.listdir(train_uri)]
      5 dataset = tf.data.TFRecordDataset(tfrecord_filenames)
----> 6 for tfrecord in dataset.take(1):
      7   serialized_example = tfrecord.numpy()
      8   stats = statistics_pb2.DatasetFeatureStatisticsList()

4 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py in __next__(self)
    759   def __next__(self):
    760     try:
--> 761       return self._next_internal()
    762     except errors.OutOfRangeError:
    763       raise StopIteration

/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py in _next_internal(self)
    745           self._iterator_resource,
    746           output_types=self._flat_output_types,
--> 747           output_shapes=self._flat_output_shapes)
    748 
    749       try:

/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/gen_dataset_ops.py in iterator_get_next(iterator, output_types, output_shapes, name)
   2726       return _result
   2727     except _core._NotOkStatusException as e:
-> 2728       _ops.raise_from_not_ok_status(e, name)
   2729     except _core._FallbackException:
   2730       pass

/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6895   message = e.message + (" name: " + name if name is not None else "")
   6896   # pylint: disable=protected-access
-> 6897   six.raise_from(core._status_to_exception(e.code, message), None)
   6898   # pylint: enable=protected-access
   6899 

/usr/local/lib/python3.7/dist-packages/six.py in raise_from(value, from_value)

DataLossError: corrupted record at 0 [Op:IteratorGetNext]

[rcrowe-google]

@alimirferdos That would be great Seyed! However, please only work in the /tfx_labs folder, since the other folders are for various other things and people might be working in them. Thanks!

[arghyaganguly]

@alimirferdos , I was able to reproduce it in the github gist.