[QUESTION]: Using appSettings.json with Azure Databricks job

Hey, I am using an appSettings.json configuration file like this in my .NET Core project

var configuration = new ConfigurationBuilder()
                .SetBasePath(Path.GetDirectoryName(Assembly.GetEntryAssembly().Location))
                .AddJsonFile("appSettings.json", false, true)
                .AddEnvironmentVariables()
                .Build();

And I have the following in my .csproj

<None Include="appSettings.json">
        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
        <CopyToPublishDirectory>Always</CopyToPublishDirectory>
</None>

This works fine running locally through spark-submit

Now I have tried following the Databricks deployment tutorial and have used the Set Jar option when configuring a new job using the publish.zip generated from dotnet publish -c Release -f netcoreapp3.1 -r ubuntu.16.04-x64 which contains my appSettings.json file along with the .dll files etc.

When trying to run this in Databricks I am getting the following error:

Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appSettings.json' was not found and is not optional. The physical path is '/databricks/driver/appSettings.json'.
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()

How can I configure Databricks so that it finds the file? The guides mention to include dependencies inside db-init.sh but I am a bit unclear of how to do that, any help is much appreciated. Thank you!

[this-fifo]

I guess I realize now uploading there would probably make it work with the Path function

My concern with using that location is that I don’t understand if that is permanent or what controls it, whereas using the root of publish.zip keeps things in my control and remove that step

Maybe that’s the solution for now?

I still would like to better understand how Databricks handle the executing assembly and why is it separate but I’m fine with the current workaround

Thank you for your help @elvaliuliuliu ! 🙇‍♂️

[elvaliuliuliu]

@this-fifo Glad to hear you have a workaround for this issue. I am not so sure but from the error above, it shows the path is /databricks/driver/appSettings.json?