Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Class `spark.SparkDataSet` not found when using 0.17.0 - fsspec (AttributeError: 'EntryPoints' object has no attribute 'get')

See original GitHub issue

Description

I started getting Class spark.SparkDataSet not found. error on 10/01/2022. Prior to this, the kedro pipeline ran fine.

Context

Kedro pipeline stop working because my datasets cannot be configured correctly.

Steps to Reproduce

Create a kedro project with a spark.SparkDataSet called core_parquet
Deploy the code to Databricks
Install kedro[spark.SparkDataSet]==0.17.0 on the Databricks cluster
Run the pipeline on Databricks

Expected Result

Dataset should have been configured correctly and the kedro pipeline should have ran fine.

Actual Result

Instead, I see the following error:

DataSetError: An exception occurred when parsing config for DataSet `core_parquet`:
Class `spark.SparkDataSet` not found.

Your Environment

Kedro version used (pip show kedro or kedro -V): 0.17.0
Python version used (python -V): 3.7
Operating system and version: Databricks 6.1 (includes Apache Spark 2.4.4, Scala 2.11)

Issue Analytics

State:
Created a year ago
Comments:10 (6 by maintainers)

Top GitHub Comments

1reaction

noklamcommented, Oct 5, 2022

@diemto Awesome! Closing the issue now, thanks for getting back with your fix.

1reaction

deepyamancommented, Oct 5, 2022

See https://stackoverflow.com/a/73932581/1093967. Can you share pip freeze and what your version of importlib-metadata is? Maybe you can pin it <5.0 for now.

Newer versions of Kedro pin importlib-metadata<5.0 (and explicitly require it), but back in 0.17.0 it’s not part of Kedro’s requirements and therefore could be unbounded.

Top Results From Across the Web

import fsspec throws error (AttributeError: 'EntryPoints' object ...

Please pin importlib-metadata<5.0.0 , this is an issue from fsspec side and the older kedro does not have an upper bound for it....

possible regression that might break other filesystems #280

I'm going to guess the error is related with the latest changes on the entry point ... AttributeError: 'EntryPoint' object has no attribute...

kedro.extras.datasets.spark.SparkDataSet

When using Databricks and working with data written to mount path points, specify filepath``s for (versioned) ``SparkDataSet``s starting with ``/dbfs/mnt .

Source code for kedro.extras.datasets.spark.spark_dataset

Built with Sphinx using a theme provided by Read the Docs. Read the Docs v: 0.17.5. Versions: latest · stable ...

Source code for kedro.extras.datasets.spark.spark_dataset

When using Databricks and working with data written to mount path points, specify ``filepath``s for (versioned) ``SparkDataSet``s starting with ...