[BUG]: Path does not exist error (despite correct path)

Describe the bug I believe I may have encountered a bug with the .Text() method (for loading from a text file into a DataFrame).

I am following the getting started tutorial: https://dotnet.microsoft.com/learn/data/spark-tutorial/run. I have all the installations completed and paths set correctly (I can successfully run other Spark apps locally).

However, for mySparkApp specifically, when I run spark-submit, I get the following error:

[Error] [JvmBridge] org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:/Users/brmurtau/mySparkApp/input.txt.

To Reproduce

I followed all of the exact steps on the .NET tutorial site, including creating a new mySparkApp, adding input.txt, and modifying Program.cs.

dotnet build runs correctly. I also set the path to my app’s dll in the assembly search environment variable (as discussed in a previous issue).

Expected behavior Logging and word count table, as shown on the tutorial at the run stage: https://dotnet.microsoft.com/learn/data/spark-tutorial/run.

Screenshots I have input.txt at the exact path that was listed in the error message:

I even tried changing the path to be full rather than relative in Program.cs but still got the same error.

Desktop:

• OS: Windows 10
• Version: Spark.NET 0.4, .NET Core 2.2

Additional context I believe this could be some sort of bug with the .Text() method not being able to find my file. I am able to successfully submit/run other Spark apps that read in CSVs. I have tried deleting and re-creating mySparkApp and input.txt, but to no avail.

[bamurtaugh]

I realize now my file name had “.txt” at the end of it, so the file was getting confused and had two “.txt” endings. Mistake I couldn’t catch at the end of a long day 😃 Issue solved!

[abhaysibal]

@bamurtaugh I ran into the same issue just now, and was stuck for long. Thanks for posting this!