[Bug] Ray Datasets schema() function should not trigger pipeline

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

Calling Ray DatasetPipeline schema() function will trigger Pipeline to be read.

• I don’t understand why schema() needs the whole pipeline to be triggered.
• This will make the below use case of Ray Dataset fail:

Create DatasetPipeline
Call DatasetPipeline.schema() to get useful schema information
Call DatasetPipeline.iter_datasets() to start iterate data        # Fail here

Error message:

  File "/Users/chongxiaoc/git/ml-code/env/py369/lib/python3.6/site-packages/ray/data/dataset_pipeline.py", line 543, in iter_datasets
    raise RuntimeError("Pipeline cannot be read multiple times.")
RuntimeError: Pipeline cannot be read multiple times.

Versions / Dependencies

Ray version: ray-2.0.0.dev0-cp36-cp36m-manylinux2014_x86_64.whl

Reproduction script

Code is heavily integrated in our infrastructure platform, I provide the pseudo code as below to reproduce, it is a common case I think.

Create DatasetPipeline
Call DatasetPipeline.schema() to get useful schema information
Call DatasetPipeline.iter_datasets() to start iterate data        # Fail here

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[ericl]

It should be cached until read yep. I think this is generally fine since the pipeline is going to get ready anyways, or deleted.

[clarkzinzow]

@chongxiaoc This would be the non-pipelined Dataset object. E.g. if you’re pipelining on an existing Dataset object, you’d get the schema from that existing Dataset object:

ds: Dataset = ray.data.read_parquet(...)
pipe: DatasetPipeline = ds.repeat(N)
schema = ds.schema()