PowerShell 7.1.3 on Raspberry Pi cannot write to writable network file

Steps to reproduce

Context is: PowerShell 7.1.3 running on Raspberry Pi 2B+ with a network mount off an Ubuntu 20.04.2 LTS box running Samba Version 4.11.6-Ubuntu

PS /home/pi> lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:        10
Codename:       buster
PS /home/pi> cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
PS /home/pi> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.1.3
PSEdition                      Core
GitCommitId                    7.1.3
OS                             Linux 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

pi@jeedom:~ $ resolveip pnjnas
IP address of pnjnas is 192.168.1.3
pi@jeedom:~ $ mount | fgrep 192.168.1.3
//192.168.1.3/pi on /pnjnas/pi type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=pi,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.3,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
//192.168.1.3/Qdownload on /pnjnas/Qdownload type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=pi,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.3,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
//192.168.1.3/public on /pnjnas/public type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=pi,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.3,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
//192.168.1.3/sto on /pnjnas/sto type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=sto,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.3,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
pi@jeedom:~ $ ls -l /pnjnas/sto/disks/Health/foo.txt
-rwxrwxrwx 1 root root 4 May  2 21:09 /pnjnas/sto/disks/Health/foo.txt
pi@jeedom:~ $ ls -ld /pnjnas/sto/disks/Health
drwxrwxrwx 2 root root 0 May  2 21:06 /pnjnas/sto/disks/Health
pi@jeedom:~ $ echo "foo" >/pnjnas/sto/disks/Health/foo.txt
# no error since file is writable
pi@jeedom:~ $ ls -l /pnjnas/sto/disks/Health/foo.txt
-rwxrwxrwx 1 root root 4 May  3 14:16 /pnjnas/sto/disks/Health/foo.txt
pi@jeedom:~ $ cat /pnjnas/sto/disks/Health/foo.txt
foo

Writing to that same file from PowerShell doesn’t work:

"foo" | out-file /pnjnas/sto/disks/Health/foo.txt

Expected behavior

The file /pnjnas/sto/disks/Health/foo.txt should be written to.

Actual behavior

Out-File: Access to the path '/pnjnas/sto/disks/Health/foo.txt' is denied.

$ ls -l /pnjnas/sto/disks/Health/foo.txt
-rwxrwxrwx 1 root root 0 May  3 14:17 /pnjnas/sto/disks/Health/foo.txt

Environment data

[jborean93]

I can replicate this myself and thought I would share more information. One thing to note that was briefly mentioned at the end is the file is created it just has no content.

What I’ve found is a few problems that cause other problems so I’ll try and break it down.

Issue 1 - Out-File fails (this issue)

Out-File fails, the file is created but is 0 length (no content).

PS /home/jborean/dev> "foo" | Out-File /mnt/test_smb/folder/foo.txt
Out-File: Access to the path '/mnt/test_smb/folder/foo.txt' is denied.

PS /home/jborean/dev> $Error[0] | Select *

PSMessageDetails      :
Exception             : System.UnauthorizedAccessException: Access to the path '/mnt/test_smb/folder/foo.txt' is denied.
                         ---> System.IO.IOException: Permission denied
                           --- End of inner exception stack trace ---
                           at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
                           at System.IO.FileStream.FlushWriteBuffer()
                           at System.IO.FileStream.FlushInternalBuffer()
                           at System.IO.FileStream.Flush(Boolean flushToDisk)
                           at System.IO.FileStream.Flush()
                           at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
                           at System.IO.StreamWriter.Flush()
                           at Microsoft.PowerShell.Commands.OutFileCommand.ProcessRecord() in
                        /PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/out-file/Out-File.cs:line 252
                           at System.Management.Automation.Cmdlet.DoProcessRecord() in /PowerShell/src/System.Management.Automation/engine/cmdlet.cs:line 173
                           at System.Management.Automation.CommandProcessor.ProcessRecord() in /PowerShell/src/System.Management.Automation/engine/CommandProcessor.cs:line 388
TargetObject          :
CategoryInfo          : NotSpecified: (:) [Out-File], UnauthorizedAccessException
FullyQualifiedErrorId : System.UnauthorizedAccessException,Microsoft.PowerShell.Commands.OutFileCommand
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:29 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 06:29 foo.txt

I’m looking at the commands mentioned in the stack trace and it seems like Out-File essentially does the following in .NET which also fails

$fs = [IO.File]::Open("/mnt/test_smb/folder/foo.txt", "Create",  "Write", "Read")
$sw = [IO.StreamWriter]::new($fs)
$sw.WriteLine("foo")
$sw.Dispose()

# Fails with
MethodInvocationException: Exception calling "Dispose" with "0" argument(s): "Access to the path '/mnt/test_smb/folder/foo.txt' is denied."
PS /home/jborean/dev> $Error[0] | Select *

PSMessageDetails      :
Exception             : System.Management.Automation.MethodInvocationException: Exception calling "Dispose" with "0" argument(s): "Access to the path '/mnt/test_smb/folder/foo.txt'
                        is denied."
                         ---> System.UnauthorizedAccessException: Access to the path '/mnt/test_smb/folder/foo.txt' is denied.
                         ---> System.IO.IOException: Permission denied
                           --- End of inner exception stack trace ---
                           at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
                           at System.IO.FileStream.FlushWriteBuffer()
                           at System.IO.FileStream.Dispose(Boolean disposing)
                           at System.IO.Stream.Close()
                           at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
                           at System.IO.StreamWriter.Dispose(Boolean disposing)
                           at CallSite.Target(Closure , CallSite , Object )
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32
                        numArgs, MemberInfo memberInfo) in /PowerShell/src/System.Management.Automation/engine/runtime/Operations/MiscOps.cs:line 2068
                           at CallSite.Target(Closure , CallSite , Object )
                           at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
                           at System.Management.Automation.Interpreter.DynamicInstruction`2.Run(InterpretedFrame frame) in
                        /PowerShell/src/System.Management.Automation/engine/interpreter/DynamicInstructions.Generated.cs:line 141
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in
                        /PowerShell/src/System.Management.Automation/engine/interpreter/ControlFlowInstructions.cs:line 358
TargetObject          :
CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : UnauthorizedAccessException
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

This is very weird because there isn’t any failure in the SMB traffic returned from the server so Samba is doing something to return access denied for some reason. There’s no attempt to write to the file, it just opens the handle, sends some set commands to change attributes but no write request. This may be an issue for the .NET team because it’s not just PowerShell.

What is even more curious is this issue seems to be related to the FileShare value. I can run the same .NET functions from above successfully if I omit the FileShare parameter or set it to None. Adding even more confusion to the pot is that Samba seems to be setting all of the FileShare values on the actual SMB open packet so something might be up with the client side share logic it has. This will require further investigation to figure out how FileShare actually operates on Linux and how .NET passes along that information to the native syscalls, i.e. .NET -> Samba.

Issue 2 - Add/Set-Content silently fails

Using Set-Content instead also has some confusing behaviour, it will not output an error but these cmdlets will create the file if it doesn’t exist but will not set the content.

PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:56 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..

PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "foo"

PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:57 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 06:57 foo.txt

PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:58 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
PS /home/jborean/dev> Add-Content /mnt/test_smb/folder/foo.txt "foo"
PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:58 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 06:58 foo.txt

So things seem to be ok but in reality the commands didn’t work.

Issue 3 - Add/Set-Content does not close the file handle

At least with the example above, the SMB open is not closed when the commands are finished. This leaves a handle still opened on the SMB server causing issues when subsequent requests go to open the file with exclusive access or the file is deleted. Using the delete example we can see that the file isn’t actually deleted until all the handles are closed (pwsh is closed/GC is collected)

PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "value"

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 07:04 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 07:04 foo.txt

PS /home/jborean/dev> Remove-Item /mnt/test_smb/folder/foo.txt

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 07:04 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 0 jborean jborean    0 May  4 07:04 foo.txt

PS /home/jborean/dev> [GC]::Collect(); [GC]::WaitForPendingFinalizers()

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 07:05 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..

You can see that the file is still present after the Remove-Item because the handle opened by Set-Content wasn’t closed and it’s only until the garbage collector has collected those handles that the file is actually deleted. Because this is dependent on when the GC runs you might get lucky and have it dispose of the handles relatively quickly but you cannot guarantee it will always run without explicitly calling it.

This can cause confusing errors if you try to then write to the file again that’s still pending a delete

PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "value"

PS /home/jborean/dev> Remove-Item /mnt/test_smb/folder/foo.txt

PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "value"
Set-Content: Could not find file '/mnt/test_smb/folder/foo.txt'.

It states it couldn’t find the file even though you want to create it with Set-Content. This is because the SMB response to opening that file returns STATUS_DELETE_PENDING because the original file still has a handle open to it (from the first Set-Content command).

It seems like there might be a stray handle in the code that Add/Set-Content calls that should be explicitly scoped with a using (...) where it is called.

[sba923]

Based on @adamsitnik’s comment in https://github.com/dotnet/runtime/issues/61175, the next steps are:

1. check if issue is fixed in 7.3.0-preview.6 when available
2. backport of the fix for https://github.com/dotnet/runtime/issues/61175 in .NET 7 to .NET 6
3. release of some 7.2.x LTS version incorporating said backport