Self-hosted v3 function app fails to run in K8S pod if filesystem is readonly

I’m hosting the Functions runtime in a dotnet Core 3.1 app to run as a Docker image inside a Kubernetes cluster. However, the function app fails early on startup with the following stack trace:

System.IO.IOException: Read-only file system
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at System.IO.Abstractions.DirectoryWrapper.CreateDirectory(String path)
   at Microsoft.Azure.WebJobs.Script.FileUtility.EnsureDirectoryExists(String path) in /src/azure-functions-host/src/WebJobs.Script/Extensions/FileUtility.cs:line 40
   at Microsoft.Azure.WebJobs.Script.WebHost.FileMonitoringService.InitializeSecondaryFileWatchers() in /src/azure-functions-host/src/WebJobs.Script.WebHost/FileMonitoringService.cs:line 192
   at Microsoft.Azure.WebJobs.Script.Utility.ExecuteAfterColdStartDelay(IEnvironment environment, Action targetAction, CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script/Utility.cs:line 728
   at Microsoft.Azure.WebJobs.Script.WebHost.FileMonitoringService.InitializeFileWatchers() in /src/azure-functions-host/src/WebJobs.Script.WebHost/FileMonitoringService.cs:line 173
   at Microsoft.Azure.WebJobs.Script.WebHost.FileMonitoringService.StartAsync(CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script.WebHost/FileMonitoringService.cs:line 94
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation, Int32 attemptCount, JobHostStartupMode startupMode) in /src/azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:line 292

Investigative information

The deployment manifest configures the pod with the following security context:

        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 10001
          runAsGroup: 10001

The function app does not host any functions at all. I tried narrowing down the root cause and it lead me to try with various triggers until I had to remove all functions from the app. However, the error still occurs.

Repro steps

Here is the Dockerfile that builds the function when creating the image:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS installer-env

COPY ./FunctionApp /src/dotnet-function-app
COPY ./nuget.conf /src/dotnet-function-app/nuget.config
COPY ./packages /src/packages 
RUN cd /src/dotnet-function-app && \
    mkdir -p /home/site/wwwroot && \
    dotnet restore --runtime linux-x64 && \
    dotnet publish --configuration Release --runtime linux-x64 --self-contained true --output /home/site/wwwroot --no-restore

FROM mcr.microsoft.com/azure-functions/dotnet:3.0
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
    COMPlus_EnableDiagnostics=0

## Run as non root user

WORKDIR /home/site/wwwroot

RUN addgroup --group app --gid 10001 \
 && useradd --uid 10001 --gid 10001 "app" \
 && chown app:app /home/site/wwwroot

USER app:app

COPY --from=installer-env ["/home/site/wwwroot", "/home/site/wwwroot"]

Please, note that the COMPlus_EnableDiagnostics environment variable MUST be set to 0 (either in the Dockerfile or in the K8S deployment manifest) otherwise, the function fails to even start with the following, cryptic error message:

Failed to create CoreCLR, HRESULT: 0x80004005

Related information

Provide any related information

• Programming language used : C#
• Bindings used: None

[jainharsh98]

@brettsam Yes, this does require documentation for the requirements and constraints when using a read-only file system with k8s for self-hosted function apps. We are currently working on validating these requirements and compiling a guidance document. Will update this thread accordingly.

[brettsam]

@lpapudippu, @jainharsh98 – this seems like a documentation issue, would you agree? Is there some official place we can document these requirements if you want to use a read-only file system?